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Executive Summary 



This first of two reports presents early findings from the National Evaluation of the 
Comprehensive Technical Assistance Centers (Comprehensive Centers), a federally funded 
program that provides technical assistance to states in connection with the No Child Left Behind 
(NCLB) Act of 2001. The law authorizing the Comprehensive Centers, the Educational 
Technical Assistance Act of 2002, mandated that a national evaluation of the program be 
conducted by the Institute of Education Sciences (IBS). The legislation indicated that the 
evaluation should “include an analysis of the services provided. . . [and] the extent to which each 
of the comprehensive centers meets the objectives of its respective plan, and whether such 
services meet the educational needs of State educational agencies, local educational agencies, 
and schools in the region.” The program evaluation is conducted by Branch Associates, Inc., 
Decision Information Resources, Inc., and Policy Studies Associates, Inc. 

This report addresses the first of the evaluation’s three rounds of data collection, 
pertaining to the Centers’ work of July 2006 through June 2007. It describes the program design 
and, drawing upon data provided by the Centers and their clients, program operations. It also 
describes assessments of Center activities and resources, reporting on quality as judged by panels 
of subject-matter experts, and on relevance, usefulness, and contributions to capacity as judged 
by practitioners (namely, state-level managers and also clients who participated directly in 
Center activities or received Center products). A final report will provide parallel findings for 
2007-08 and 2008-09. In addition, it will present findings from case studies of capacity building 
at the state level and any changes in findings over time. 

The main findings from the evaluation so far are: 

■ The Comprehensive Centers reported planning their work in coordination and 
consultation with their clients with the work evolving during the year. All 16 

Regional Comprehensive Centers (RCCs) reported obtaining state input into their 
initial plans and engaging states in refinements to the plans through ongoing 
interaction and negotiation. Similarly, all five Content Centers (CCs) reported 
forming their work plans incorporating RCC input acquired through either RCC 
staff surveys or direct communication. In addition, all five CCs described working 
with the U.S. Department of Education (ED) to learn of specific topics and tasks 
needed to advance ED priorities. A review of projects conducted indicates that both 
RCCs and CCs adjusted their work plans during the year. 

■ The technical assistance activities were varied and consisted of ongoing 
consultation, research syntheses, planning of technical assistance with 
participants, training events, conferences, and support for task forces or for 
development of formal plans. Consistent with the mission of “front-line” 
assistance, the majority of sampled RCC projects involved ongoing consultation and 
follow-up (82 percent). CC assistance most often focused on the delivery of research 
information, consistent with the CCs’ prescribed focus on synthesizing, translating, 
and delivering knowledge on a particular topic. The delivery of research collections 
and syntheses occurred in 74 percent of the CC sampled projects. 
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■ The Comprehensive Centers program delivered technical assistance that, 
according to state managers: (1) served state education agencies’ (SEAs’) 
purposes in seeking technical assistance, (2) was aligned with SEAs’ 
priorities for NCLB-related technical assistance, and (3) was perceived to 
expand SEA capacity. Eighty-eight percent of state managers rated the technical 
assistance they received from Centers as at least “a good start” in serving their 
purposes, and 36 percent overall reported that it “served the state’s purposes 
completely.” For each of the four areas of NCLB implementation most widely 
identified as state priorities for technical assistance, at least 90 percent of those 
state managers who had identified the area as a priority had received assistance 
with it from the Centers. Overall, more than two-thirds of state managers (68 
percent) reported a perception that assistance from the Comprehensive Centers 
had greatly expanded their state’s capacity to carry out its responsibilities in at 
least one NCLB area. 

■ Center projects in the evaluation sample were judged hy clients to he on 
average in the “moderate” to “high” range of relevance and usefulness; 
panels of experts judged their quality to he in the “moderate” range on 
average. On a scale of 1 to 5 with a 3 representing “moderate” and a 4 
representing “high,” the programwide average ratings for the sampled projects 
were 3.34 for technical quality (scored by panels of content experts), and 3.94 for 
relevance and 3.70 for usefulness (scored by participants).' The average quality 
rating was higher among CCs than RCCs by more than one -half of a standard 
deviation; the average relevance rating was higher among RCCs than CCs by at 
least one-half of a standard deviation; usefulness ratings were similar between the 
two Center types (i.e., did not differ by at least one -half of a standard deviation).^ 



* This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 

^ Using Cohen (1988) as a conceptual framework, we estimated Cohen's d (an estimate of the effect size defined as 
the difference in means divided by the pooled standard deviation) and adopted the logic of Cohen for what is a 
moderate difference. Specifically, we adopted a difference in the means of one-half of one standard deviation 
(analogous to an effect size of .5) as our minimum threshold for highlighting differences. The “pooled standard 
deviation” for each computation varied with the unit of analysis. For analyses conducted at the Center level, the 
pooled standard deviation was computed as the standard deviation of the variable of interest (e.g., relevance) 
computed at the Center level. For analyses using the project as the unit of analysis, the pooled standard deviation 
was computed at the project level. 
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The Comprehensive Centers Program 



In its authorization, the Comprehensive Centers program was given an overall eharge of 
supporting state and loeal NCLB implementation. ED, using discretion provided in the 
legislation, established two major program features that differed from past Comprehensive 
Centers programs:^ 

■ First, the primary focus would be on assisting states to expand and strengthen 
states’ capacity to deliver assistance to schools and districts; ED specified that 
Centers could only work directly with districts or schools under special 
circumstances. 

■ Second, awards would be made in two tiers, to 16 RCCs and 5 CCs. They were 
instructed to work as follows: 

■ Each RCC was charged with providing “frontline assistance” either to one 
large state or to a group of two to eight states and other jurisdictions."^ The 
RCCs were also expected to deliver technical assistance to their assigned 
states, addressing the needs and building capacity of the states to assist 
their districts and schools. 

■ Meanwhile, each CC would work on a nationwide basis within a particular 
substantive area: Assessment and Accountability, Instruction, Teacher 
Quality, Innovation and Improvement, or High Schools. CCs would 
facilitate access to, and use of, existing research and practices. 

■ The absolute priorities for the two types of Centers indicated that they 
should work together: Regional Centers should draw information and 
resources from Content Centers as well as other sources; and Content 
Centers should both supply knowledge to Regional Centers and “work 
closely with Regional Centers to provide technical assistance to States.” 



Evaluation Topics and Methods 

The research priorities for the evaluation were primarily driven by the statute and focused 
on the following key research questions: 

■ What are the objectives of the Comprehensive Center network and of each Center? 



^ Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 
32583-94. 

The nonstate jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto Rico, 
the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of 
Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, Republic of the Marshall Islands, and Republic of Palau. 
Throughout this report, the term “state” will be defined to include the 50 states as well as these other jurisdictions. 



XV 




What kinds of products and services are provided by the Comprehensive Center 
network and by eaeh Center? 



■ How do Centers develop, refine, and earry out their plans for technieal assistance? 
How do they define their elients’ educational needs and priorities? How do Center 
clients (states or Regional Centers) define their needs and priorities? 

■ To what extent is the work of eaeh Comprehensive Center of high quality, high 
relevance, and high usefulness? 

■ To what extent do states report that Center projects have expanded state eapacity 
to address underlying needs and priorities and meet the goals of NCLB? 

■ To what extent have states relied on other sourees of teehnieal assistance besides 
the Comprehensive Centers? What other sources? How does the usefulness of 
Center projeets compare with the usefulness of projeets from other sourees? 

The evaluation gathered annual information from six data sourees in order to address the 
researeh questions above. Data colleetion included; 

■ Management plans. The evaluation reviewed these as a data source for eaeh 
Center’s intended foeus at the beginning of the year, drawing from the plans a list 
of topies as foei of Center objectives. 

■ Center staff interviews. Using struetured response eategories. Center staff were 
asked about how they planned their programs of work; how their plans evolved 
during the program year; and what they offered to elients with respect to the 
topies addressed, the delivery modes used, and their sourees for eontent expertise. 
(See appendix C for the protoeols and other structured response materials used 
during the interviews). 

■ Survey of senior state managers. SEA managers were surveyed about their 
state’s teehnieal-assistanee needs and what the Centers (ineluding their RCC and 
the CCs) had provided. 

■ Project inventory forms. The evaluation team assisted eaeh Center in grouping 
related activities and deliverables into “projeets,” with the project defined as a 
group of closely related activities and/or deliverables designed to achieve a 
specific outcome for a specific audience. Projeets were in turn classified by the 
Centers into major, moderate, and minor projects on the basis of the relative level 
of effort they refleeted. The Centers and the evaluation team also classified the 
projeets, according to the topies addressed, into 22 topical categories.^ 



^ The 22 topics were: components of effective systems of support for states, districts, and schools; data use or data- 
driven decision making; formative assessment; reading; adolescent literacy; mathematics; dropout prevention; high 
school redesign or reform; transition to high school; special education curriculum, instruction and professional 
development; special education assessment; English language learners;” highly qualified teacher” provisions of 
NCLB; teacher preparation and induction; teacher professional development; supplemental educational services; 
Response to Intervention; migrant education; Indian or Native American education; data management and 
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■ Survey of project participants. A representative sample of clients who had 
participated directly in the evaluation’s purposive sample of major and moderate 
Center projects furnished descriptive information, through surveys, on the 
technical-assistance needs of their offices and on the activities and resources that 
the project had delivered to them. These clients included individuals working at 
the state level who had participated in RCC or CC projects and RCC employees 
who were among the clients of CC projects. They also rated the relevance and 
usefulness of the sampled projects. 

■ Expert panel review. The same sample of major and moderate projects was 
reviewed for quality by a panel of experts. Content experts were recruited and 
trained to use standard criteria to rate the technical quality of the sampled Center 
projects on the basis of a review of all project materials. 



Operation of Centers 

Before the beginning of the 2006-07 program year, each Center was required to submit a 
management plan, setting out objectives and planned activities for the coming year, for ED 
review and approval. Almost all Centers (20 of 21) reported that client input was used in writing 
the plan. The same number reported conducting needs assessment through meetings or other 
communication with clients. 

In addition to initial needs assessments, to ultimately meet client needs Centers used 
planning and ongoing interactions with each other and with the SEAs to refine their needs 
assessments as indicated in exhibit ES.l. Eifteen of 16 RCCs reported forming work groups 
within state organizations that brought together staff from multiple departments to discuss service 
needs and delivery; the remaining RCC was 1 of 14 that reported working directly with the chief 
state school officer in their initial planning. Half of the RCCs (8) formed cross-agency work 
groups to discuss SEA service needs and delivery. All five CCs identified needs of their client 
RCCs primarily by conducting conference calls with designated RCC representatives, and three of 
the five CCs maintained communication about needs by forming workgroups that included RCC 
representatives. In addition, all CCs reported providing either large-group events or support of 
existing RCC programs and projects as additional ways to learn about and meet client needs. 



compliance; assessment design; and parent involvement. In addition, projects that addressed none of these 22 topics 
were categorized as “other.” 




Exhibit ES.1. Center strategies for pianning and ongoing interactions with 
clients 



Center strategy 


RCCs 

(N=16) 


CCs 

(N=5) 


All 

(N=21) 


Assess RCC needs through meetings or other communication 
with RCC staff 




5 


- 


Included SEA input when writing annual management plans 


16 


- 


- 


Sponsor large events to make contact with many clients 


13 


4 


17 


Form work groups within client organizations that bring together 
staff from multiple departments/divisions to discuss service 
needs and delivery 


15 


1 


16 


Offer service to support existing client programs/ projects/ 
policies 


9 


4 


13 


Form work groups across client organizations to discuss service 
needs and delivery (e.g., different SEAs and/or RCCs) 


8 


3 


11 



EXHIBIT READS: All five CCs assessed RCC needs through meetings or other communication with RCC 
staff. 

SOURCE: Verbatim summaries of Center interviews conducted during 2007 site visits, coded by evaluation 
team, with Center review of coding results 



Although Centers reported providing teehnicai assistance in a majority (80 percent) of the 
instances where identified topic areas were included in their management plans, Centers 
commonly adjusted the topic areas in which they conducted work, adding work in some areas 
and shifting away from work in other areas. Based on a review of each Center’s inventory of 
projects against the topics that had been included in that Center’s management-plan objectives, 
there were instances of the Center carrying out work on a topic not initially cited in its 
objectives. That is, in 19 of 22 topic areas Centers reported delivering technical assistance that 
was not in their original management plan. The most common shift toward topics were in the 
areas of Response to Intervention (seven Centers conducted work in this area that was not 
planned), English language learner issues (five Centers), highly qualified teacher provisions of 
NCLB (five Centers) and supplemental educational services (five Centers). By the same token. 
Centers appeared to delete work in particular topic areas such as special education curriculum, 
instruction, and professional development (five Centers appeared to delete planned work); and 
data use or data-driven decisionmaking (four Centers appeared to delete planned work). Centers 
that set an objective in a topic did not report a project on the topic in their inventory for 10 of the 
22 topic areas. The most common topic area for projects was that of statewide systems of support 
for educational improvement. 

Regardless of whether they were originally specified in work plans or added later, an 
analysis of the projects that the evaluation team sampled for closer study across all Centers 
provides more in-depth information about the nature of Center technical assistance activities or 
resources. Although the sample of projects is not statistically representative of the Centers’ work, 
the process of sample selection favored each Center’s most dominant projects and included over 
half (56 percent) of that year’s designated major or moderate projects. Most projects (84 percent) 
used more than one mode of delivery from a list that included conferences, training, delivery of 
research collections or syntheses, support for a task force, support for development of a plan or 
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policy, engagement of clients in projeet planning, and ongoing consultation and follow-up. 
Across all the sampled projeets and also aeross the Regional Centers’ sampled projects, the most 
frequent modes of delivery as shown in exhibit ES.2 were ongoing consultation and follow-up 
(84 of 122, or 69 percent of all projects and 79 of 96, or 82 percent of Regional Center projects) 
or delivery of a researeh colleetion or synthesis (71 of 122, or 58 pereent of all projects and 52 of 
96, or 54 percent of Regional Center projects). The Content Centers’ projeets most often 
included delivery of a research collection or synthesis (20 of 27, or 74 pereent of their projects) 
or a conference (17 of 27, or 63 percent). 

While some projects were worked on by both the RCCs and the CCs, coordination 
between CCs and RCCs when it did occur was asymmetrical. In providing assistance to states, 
RCCs used CC input more than CCs used substantive RCC input. Almost half of the sampled 
RCC projeets had a substantive CC eontribution (sueh as a product or a presentation by a CC 
staff member); in contrast, in 1 1 percent of the sampled CC projects an RCC contributed eontent 
or delivered assistance. More often (in 37 percent of the sampled CC projects), the CC enlisted 
the help of one or more RCCs to identify and reeruit partieipants. 



Exhibit ES.2. Sampied Center projects by types of participant activities 
and products 



Activities and products 

(with clarifying definitions used by coders) 


RCC 

projects 

(n=96) 


CC 

projects 

(n=27) 


Aii 

projects 

(n=122) 


Ongoing consultation and follow-up (multiple contacts to same 
participants, that were part of a coherent and purposeful whole) 


79 


6 


84 


Research collections and syntheses 


52 


20 


71 


Engagement of participants in project planning (more than needs 
assessment or identifying participants) 


43 


8 


50 


Training events (focused on implementing a specific program or 
strategy) 


41 


10 


50 


Task force meetings and work (focused on addressing a specific 
problem, program, or policy) 


48 


2 


50 


Conferences (symposium, forum, institute; highlights a range of 
perspectives, strategies, or programs) 


26 


17 


43 


Support development of a formal plan to implement a program or 
policy 


18 


2 


20 



EXHIBIT READS: Seventy-nine RCC projects included ongoing consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials; cover sheets 
coded by evaluation team. The total number of projects was 122. One project collaboratively conducted by an 
RCC and a CC was counted among both RCC projects and CC projects but was only counted once among the 
projects of all Centers. 
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The types of work emphasized in the RCC and CC sampled projeets were eonsistent with 
their different eharges. RCC assistanee more often ineorporated sustained interaction with 
participants: again, the majority of RCC projects in the sample involved ongoing consultation 
and follow-up (82 percent), whereas this was less so for the work of the CCs (22 percent of 
projects in the sample). Thus, this pattern of RCC activities was consistent with the mission of 
“front-line” assistance that would take clients’ purposes and circumstances into account and 
provide ongoing support for their implementation of NCLB. For the CCs, the assistance more 
often focused on the delivery of research information, consistent with the CCs’ prescribed focus 
on synthesizing, translating, and delivering knowledge on a particular topic. The delivery of 
research collections and syntheses occurred in 74 percent of the CC projects but 54 percent of the 
RCC projects. The sampled CC projects more often delivered technical assistance through 
conferences (63 percent of the CC projects but 27 percent of the RCC projects). 



Ratings of Center Assistance 

The sampled projects, all identified by the Centers as “major” and “moderate,” were rated 
in order to assess the services provided by the Comprehensive Centers program. Each project 
was evaluated for relevance and usefulness by a sample of participants — state staff, intermediate 
agency staff, local educators working on behalf of the state, and RCC staff — who were the 
intended beneficiaries of the project and who had received at least some of the technical 
assistance it provided. Ratings of project quality were gathered from panels of experts with 
strong knowledge of the content or substantive focus of the specific projects they reviewed. 
Relevance was assessed with eight survey items and usefulness with 1 1 items; quality was 
judged on three items called dimensions (exhibit ES.3). Each overall measure (relevance, 
usefulness, or quality) was calculated as the mean of ratings assigned to each item. The item- 
level ratings themselves were based on 5-point rating scales.^ 



^ Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 
levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 
set of items within the three measures and the different contexts for the ratings (experts who underwent training for 
the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 
different subsets of project activities), the ratings across the three measures are not directly comparable. 
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Exhibit ES.3. 



Reievance, usefuiness, and quaiity items 



From expert panel scoring From project participant surveys 

Technical quality Relevance Usefulness 



Reviewers were directed to 
assign a score to each 
dimension and to include the 
basis for their ratings on the 
rating form, including the specific 
artifacts on which their score was 
based. The three dimensions 
are: 

a. Demonstrated use of the 
appropriate documented 
knowledge base - to include 
an accurate portrayal of the 
current state of information 
with prominence to those 
with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 
knowledge base to the 
products and services 
provided - materials are 
consistent with the 
best/accurate information 
available and the 
presentation adequately 
conveys the confidence of 
the information 

c. Clear and effective delivery - 
information is well organized 
and written and accessible to 
the intended audience for 
easy use 



Based on your experience, to 
what degree was this set of 
activities and resources relevant 
to your work, in each of the 
following respects? 

a. Addressed a need or problem 
that my organization faces 

b. Addressed an important 
priority of my organization 

c. Addressed a challenge that 
my organization faces related 
to the implementation of NCLB 

d. Provided information, advice, 
and/or resources that could be 
directly applied to my 
organization’s work 

e. Addressed our particular state 
context 

f. Addressed my organization’s 
specific challenges (e.g., 
policy environment, leadership 
capacity, budget pressures, 
local politics) 

g. Provided information, advice, 
and/or resources that could be 
used to guide decisions about 
policies, programs, or 
practices 

h. Highlighted the implications of 
research findings (or 
information about best 
practice) for policies, 
programs, or practices 



Based on your experience, to 
what degree was this set of 
activities and resources useful to 
you, in each of the following 
respects? 

a. Provided resources that were 
easy to understand and easy 
to use 

b. Employed an appropriate 
format (e.g., a work group, a 
conference, individual 
consultation, written products) 

c. Provided adequate opportunity 
to learn from colleagues in 
other states 

d. Included adequate follow-up to 
support the use of new 
information and resources 

e. Were timely 

f. Helped my organization solve 
a problem 

g. Helped my organization 
maintain or change a policy or 
practice 

h. Helped my organization take 
the next step in a longer-term 
improvement effort 

i. Provided my organization with 
information or resources that 
we will use again 

j. Helped my organization 
develop a shared expertise or 
knowledge-base 

k. Helped individuals in my 
organization to develop skills 
that they will use again 



Based on the ratings, Center teehnieal assistanee was judged to be in the “moderate” to 
“high” range of quality, relevanee, and usefulness. On a seale of 1 to 5 with a 3 representing 
“moderate” and a 4 representing “high,” the programwide average ratings for the sampled 
projeets were 3.34 for teehnieal quality (seored by panels of eontent experts), and 3.94 for 
relevanee and 3.70 for usefulness (seored by partieipants) as indieated in exhibit ES.4.’ 



^ This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 
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Exhibit ES.4. 



Center Levei Mean ratings of technicai quaiity, reievance, and 
usefuiness 





Technical 

quality 


Relevance 


Usefulness 


All Comprehensive 
Centers (N=21) 


3.34 


3.94 


3.70 


All RCCs (N=16) 


3.21 


3.99 


3.71 


All CCs (N=5) 


3.73 


3.78 


3.65 


Difference of RCC and 
CC means 


-0.52^ 


0.21^ 


0.06 


Pooled standard deviation 
(all Comprehensive 
Centers) 


0.41 


0.34 


0.34 


Ratio of difference in means 
to pooled standard deviation 


-1.28 


0.60 


0.18 



NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 
ratings for the three quality dimensions. A notation of ^ indicates that the difference in the mean ratings between the 
CCs and RCCs is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 
ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 
ratings. 



Given that the RCC and CC roles and aetivity emphasis differed, the evaluation looked at 
variation aeross Center types and projects that might provide information for program 
improvement. The average quality rating was higher among CCs than RCCs by more than one- 

o 

half of a standard deviation ; the average relevance rating was higher among RCCs than CCs by 
at least one -half of a standard deviation; usefulness ratings were similar between the two Center 
types (differing by less than one -half of a standard deviation). The Content Centers received 
Center-level mean scores for technical quality that averaged 3.73, compared with 3.21 for the 
Regional Centers; the difference of 0.52 points exceeded one-half of one pooled standard 
deviation. The mean scores for relevance were 3.99 for the Regional Centers and 3.78 for the 
Content Centers. On usefulness, the mean score of 3.71 for the RCCs and 3.65 for the CCs were 
within one-half of a standard deviation of each other. 

There was variation in the ratings across and within individual Centers. On each measure, 
at least 1 1 Centers had a mean rating that was at least one-half of a standard deviation above or 
below the overall mean for its type of Center (RCC or CC) for that measure^ (i.e., 11 of 21 



* For analyses conducted at the Center level, the pooled standard deviation was computed as the standard deviation 
of the variable of interest (e.g., relevance) computed at the Center level. 

^ For analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project 
level. 
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Centers were this far above or below the mean for quality, 1 1 for relevanee, and 14 for 
usefulness). One RCC was rated higher than others by at least one -half of a standard deviation on 
all three measures, and one CC and one RCC were rated lower than others on all three measures. 
Aside from these Centers, the other 18 Centers’ ratings were not eonsistently higher or lower 
than the mean but varied aeross measures (Exhibit ES-5). 

Exhibit ES.5. Mean ratings of technicai quaiity, reievance, and usefuiness, by 
Center 



Center type 


Technical 

quality 


Relevance 


Usefulness 


RCCs (N=16) 


3.78t 


3.784 


3.424 




3.63t 


3.224 


3.004 




3.46t 


4.18t 


3.94t 




3.44t 


3.90 


3.63 




3.36 


3.97 


3.63 




3.35 


3.97 


3.57 




3.35 


3.93 


3.514 




3.31 


4.15 


3.69 




3.21 


4.08 


3.82 




3.17 


4.3lt 


4.05t 




3.15 


4.12 


3.93t 




3.11 


4.70t 


4.46t 




2.984 


4.07 


3.92t 




2.744 


3.204 


3.054 




2.744 


4.01 


3.54 




2.634 


4.18t 


4.17t 


Average RCC rating 


3.21 


3.99 


3.71 


Pooled standard 


deviation (RCCs) 


U.32 


U.-if 


U.33 


CCS (N=5) 


4.24t 


3.76 


3.544 




3.94 


3.90t 


3.86t 




3.88 


3.99t 


3.84t 




3.444 


3.584 


3.444 




3.144 


3.684 


3.56 


Average CC rating 


3.73 


3.78 


3.65 


Pooled standard ... ... ... 


deviation (CCs) 


U.4J 


0.7b 


0.79 



NOTE: The arrow pointing upward indicates the accompanying value is at least one-half of one standard 
deviation above the group mean (e.g., 3.78 is at least one-half of one standard deviation above the mean 
for the RCCs). The arrow pointing downward indicates the accompanying value is at least one-half of 
one standard deviation below the group mean. 

EXHIBIT READS: One of the RCCs had a mean rating for technical quality of 3.78, a mean rating for 
relevance of 3.78, and a mean rating for usefulness of 3.42, across the projects sampled from that 
Center. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 
participants for relevance and usefulness. Responses weighted so that each panelist or participant 
contributed equally to project ratings, and each project contributed equally to Center ratings 
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The evaluation also looked at the relationship between the three measures: quality, 
relevanee, and usefulness. It was reasoned that the eontent experts rating quality and the 
partieipants rating relevanee and usefulness might value and be better able to judge different 
qualities in a Center project, which is why we did not have content experts evaluate the projects 
for their utility or the participants assess the technical quality. An examination of the associations 
among the three dimensions was conducted by calculating correlation coefficients. Such a 
statistic indicates the strength and direction of a linear relationship between two factors. A 
correlation coefficient can vary from positive one (indicating a perfect positive relationship), 
through zero (indicating the absence of a relationship), to negative one (indicating a perfect 
negative relationship). If the correlation is statistically significant (p <.05), we can have strong 
(95 percent) confidence that what we calculated is not due to chance. 

Ratings of quality were unrelated to ratings of relevance and usefulness, although 
relevance and usefulness ratings were highly correlated with each other. The correlation 
coefficient for relevance and usefulness was +0.84, while the coefficient of relevance with 
quality was -0.12, and the coefficient of usefulness and quality was -0.04. In other words, the 
extent to which a project faithfully reflected the knowledge base on a topic and provided 
appropriate caveats about the quality of its evidence was unrelated to the extent to which 
participants deemed that project relevant or useful to their agency. 

Given the variation in ratings across projects, additional analyses of project 
characteristics were conducted to explore whether there were any consistent patterns between 
ratings and the particular features of the projects. Such information may provide suggestions for 
possible program improvement. Specifically, if there is a consistent relationship between scale of 
the undertaking and the ratings, perhaps signaling more ambitious projects or projects that allow 
a greater focus of Center resources on the effort, then this might be suggestive of productive uses 
of Center resources for future emphasis. In fact, projects identified by the Centers as “major” 
were rated higher by at least one-half a standard deviation on the measures of relevance and 
usefulness but not on the measure of quality. 

Projects with particular types of activities may be easier to carry out, may play to Center 
strengths, or may be seen as more productive to the ultimate clients. In addition, those RCC 
projects that included CC contributions might be expected to have higher quality ratings than 
other RCC projects, given the expected content and research focus of the CCs. Thus, the 
evaluation compared ratings of subgroups of projects (e.g., those with and without particular 
activities such as conferences, training, or research syntheses; RCC projects with and without CC 
involvement) to see if there were any consistent relationships between the ratings and particular 
Center activities or the incorporation of CC work. Across five of the seven project activities 
identified, comparisons of projects with and without the activities showed no differences in 
quality, relevance, or usefulness greater than one-half of a standard deviation. Ratings of RCC 
projects with CC contributions did not differ by more than one -half of a standard deviation on 
any measures, compared with those without CC contributions. Thus, these analyses do not 
suggest differences in ratings related to CC contributions or particular activities. 



*** For this analysis, the evaluation team used Spearman’s rank order correlation, as this non-parametric rating is the 
appropriate statistical function to describe correlations between two variables where the values of the variables are 
not normally distributed and are on a scale (such as ratings). 
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It is also possible that Center teehnieal assistanee is viewed as more benefieial by some 
types of partieipants or that the extent to whieh partieipants are engaged in a projeet affeets how 
they regard it. Therefore, the ratings of subgroups of individuals were examined for the relevanee 
and usefulness measures. Involvement with the projeet design and time spent in projeet aetivities 
were both assoeiated with statistieally higher relevanee and usefulness ratings. There was also 
evidenee of statistieally higher ratings among respondents whose job had a foeus on NCLB- 
related responsibilities (defined as respondents who spent at least 25 pereent of their time on the 
job on NCLB). This suggests that those who worked on the projeets and were most likely to 
benefit from the work rated the projeets higher. 



State Capacity Building and the Use of Different Sources of 
Technical Assistance 

Capaeity building was prominent as a goal for the Comprehensive Centers program. The 
first priority for all Centers, artieulated by ED in the Notiee Inviting Applieations, ineluded 
“helping states build the eapaeity to help school districts and schools implement NCLB 
provisions and programs.”*' 

Fifty-three percent of state managers reported that technical assistance from the program, 
including both their Regional Center and any Content Centers with which they had experience, 
had expanded state capacity to a “great extenf ’ or “very great extent” for building or managing a 
statewide system of support. This was the area of NCLB responsibility in which extensive 
capacity building was most widely reported. In addition, the Centers were the top source used for 
help “to plan the initial steps in solving a problem,” reported by 66 percent of state managers, 
and “to develop the skills of SEA or intermediate education agency staff,” reported by 61 percent 
of state managers. A case study component of the evaluation will examine further the 
contribution of the Comprehensive Centers to building state capacity. 

The Comprehensive Centers were one resource among several available to, and used by, 
state managers. On average, state managers ranked the Centers as one of the top three sources of 
technical assistance that they relied upon, along with professional associations and the ED- 
funded Regional Educational Laboratories. The Centers were not the resource used most widely 
for “working with districts and schools,” a purpose that ED de-emphasized in the Centers’ 
charge: colleges and universities were used for this purpose by 37 percent of state managers, and 
consulting firms by the same percentage; the Centers were used for this purpose by 22 percent of 
state managers. 



Summary and Next Steps 

This evaluation addresses questions about the technical assistance provided by the two 
types of Comprehensive Centers; how the Centers work with their clients; the match between 
client purposes and assistance delivered; and assessments of the quality, relevance, and 
usefulness of a sample of technical assistance projects. This interim report presents findings from 



*' Notice Inviting Applications, 32585 . 
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2006-07, the Centers’ second year of operation under a new design that ED established in 2005. 
The findings suggest that Centers attempted to strike a balance between adhering to their 
management plans for the year and accommodating client requests, and that state clients 
generally viewed Center technical assistance as serving state purposes. In addition, the study 
found that RCCs and CCs worked in the different ways that had been mandated in the design of 
the two types of Centers. 

In this first round of project ratings, mean ratings across sampled projects and all Centers 
fell in the “moderate” to “high” range for quality (rated by expert panels) and relevance and 
usefulness (rated by participating clients). The CCs had higher mean ratings of technical quality 
for their sampled projects than did RCCs, while the RCCs had higher mean ratings of relevance 
than did CCs. There was no statistically significant relationship between ratings of quality on the 
one hand and relevance or usefulness on the other. These findings suggest that at least in the 
2006-07 program year, achieving high technical quality was unrelated to delivering assistance 
that clients found highly relevant or useful. 

The evaluation team is continuing to study Center operations, outputs, and outcomes for 
the 2007-08 and 2008-09 program years. By repeating the processes of expert panel reviews and 
surveys, the evaluation team will be able to report on changes over time in the quality, relevance, 
and usefulness of Center projects as well as on trends in state managers’ perspectives on Center 
technical assistance and contributions to capacity building in SEAs. 
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1. Introduction and Background 



The Educational Technical Assistance Act of 2002 authorized the Secretary of Education 
to award “not less than 20 grants to local entities, or consortia of such entities, with demonstrated 
expertise in providing technical assistance and professional development in reading, 
mathematics, science, and technology” (Section 203). The law mandated that these entities, 
called Comprehensive Technical Assistance Centers, undergo independent evaluation under the 
direction of the National Center for Education Evaluation and Regional Assistance (NCEE) in 
the Institute of Education Sciences of the U.S. Department of Education (ED). 

This is the first interim report of a multi-year evaluation conducted for NCEE by Branch 
Associates, Inc., Decision Information Resources, Inc., and Policy Studies Associates, Inc. The 
report addresses the second year of operation of the Center system, July 2006-June 2007. This 
introductory chapter provides background information on ED’s purposes and design for the 
Center system. It also describes key state responsibilities included in the No Child Eeft Behind 
(NCEB) Act with which the Centers were expected to provide assistance. The chapter concludes 
with a description of the major evaluation questions. The study methods are described in the 
second chapter of this report, followed by the findings in successive chapters. 



The Center Program 

The current Centers replaced 15 existing Comprehensive Regional Assistance Centers 
established under the Elementary and Secondary Education Act of 1994 (ESEA). The current 
system represented a departure from that previous system with respect to the targeting of services 
and the responsibilities of Centers. 

The 1994 reauthorization of ESEA had charged the previous set of 15 Comprehensive 
Centers with delivering assistance to support standards-based reform as envisioned in other 
sections of the 1994 law. They were to provide training and technical assistance to states, local 
education agencies (EEAs), schools, tribes, community-based organizations, and other ESEA 
grantees related to several areas of local responsibility. These included: (1) improving the quality 
of instruction, curricula, assessments, and other aspects of school reform; (2) implementing 
effective schoolwide programs; and (3) meeting the needs of children, especially children in 
high-poverty areas, migrant children, immigrant children, limited-English-proficient children 
(EEP), neglected or delinquent children, homeless children, Indian children, and children with 
disabilities [Section 13102 (a)(l)(A-E)]. In short, the previous Centers’ mandate included a focus 
on a number of aspects of local educational practice. 

A previous evaluation of the Comprehensive Centers,^ based on surveys of clients 
conducted in 1999, found that the majority of direct participants in major Center training and 



* U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 
Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 
Evaluationiy ohxme. I). Washington, DC: Author. 
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technical assistance activities were school staff (either teachers or principals), and that the majority 
of clients who arranged for services from the Comprehensive Centers were also based in school 
districts or schools. At the state level, among those state education agency (SEA) staff members 
who had received assistance, 64 percent reported that the Centers had improved the ability of their 
SEA to provide assistance to districts and schools. The report noted that higher ratings for Centers 
were associated with more intensive technical assistance, based upon survey findings. 

The Educational Technical Assistance Act of 2002 authorized a new group of 
Comprehensive Centers to provide technical assistance for NCEB implementation at the state, 
district, substate region, and school levels for the purposes of “improving academic achievement, 
closing achievement gaps, and encouraging and sustaining school improvement” (Section 203). 

It also gave ED discretion to “establish priorities” for the Centers (Section 207). 

The current design of the Comprehensive Centers program reflects changes from the 
design of the predecessor program. In making its design choices, ED set up an advisory process 
to identify priorities for the new Centers: in 2004 the Secretary of Education appointed 10 
Regional Advisory Committees that would conduct needs assessments in their regions and make 
recommendations regarding technical assistance. The committees said SEAs needed help making 
better use of scientifically based research in decisionmaking, and that strengthening SEAs’ 
capacity to serve local school districts was critical to the success of NCEB reforms, according to 
a synthesis of their recommendations. 

hollowing this effort, ED identified states as the Centers’ primary client base, although 
the program had in the past served local clients, as described above. ED charged the Centers to 
work in new ways to expand and strengthen states’ capacity to deliver assistance to schools and 
districts. The new program would also consolidate and expand responsibilities for 
Comprehensive Center technical assistance in that it would replace the Regional Technology in 
Education Consortia, the Eisenhower National Clearinghouse for Mathematics and Science 
Education, and the Regional Mathematics and Science Education Consortia. 



Structure of the Comprehensive Center System 

To implement the provisions of the Educational Technical Assistance Act of 2002, ED 
created a new, two-tiered Center system. The Notice Inviting Applications detailed the design of 
that system. Under the new system, the Secretary of Education would award grants to 21 
Centers, each tasked with “provid[ing] technical assistance to States as States work to help 
districts and schools to close achievement gaps in core content areas and raise student 
achievement in schools. To accomplish this goal, ED stipulated that applicants had to “propose a 
plan of technical assistance specifically focused on helping States implement the provisions of 
NCEB applicable to States, and helping States build the capacity to help school districts and 
schools implement NCEB provisions and programs.”^ 



^ Sheekey, A., Cymrot, D.J., and Fauntleroy, C. (2005, March). A Report to the U.S. Department of Education: 
Overview and Synthesis of the Regional Advisory Committee Reports on Educational Challenges and Technical 
Assistance Needs. Alexandria, VA: The CNA Corporation. 

^ Ibid, 32585. 
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While the overall goal of assisting states with NCLB implementation was common to all 
21 Centers, the two-tiered system designed by ED created distinct roles for the two types of 
Centers. These are described next. 



Distinct RCC and CC Roles 

Within the 21 Centers, ED constituted 16 Regional Comprehensive Centers (RCCs) and 
five Content Centers (CCs). By design, RCCs and CCs were given different roles and functions 
in a system of NCLB technical assistance (exhibit 1.1). RCCs, embedded within distinct 
geographic regions across the United States and territories, would deliver technical assistance to 
the states and territories in their region, addressing their needs and building their capacity to 
assist their districts and schools. Meanwhile, each of the CCs would take responsibility for 
synthesizing knowledge from the research and promising practices within a particular 
substantive area. The CCs would build the capacity of the RCCs by providing research-based 
information, products, guidance, and knowledge on key topics. The CCs would also work with 
RCCs to provide technical assistance to states. In turn, each state would help its districts and 
schools meet NCLB requirements. 

Exhibit 1 .1 . Center network design 




Elaborating on the functions of RCCs, ED required them to work directly with states to “provide 
frontline assistance.”* ED mandated that RCCs provide states with ongoing assistance and 
training that would draw from a range of knowledge sources, including but not limited to CCs; 
provide CCs with information about promising practices; convene states for collaboration; 



' Ibid., 32585. 
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and deliver information based on both researeh and best praetiee. The speeific list of RCC 
responsibilities ineluded the following:^ 

■ “Working closely with each State in its region on an ongoing basis” 

■ “Linking States with the resources of Content Centers, Department staff, Regional 
Educational Laboratories, The What Works Clearinghouse, and other entities” 

■ “Suggesting sources of appropriate service providers or assistance for State 
activities that are not within the core mission of the centers” 

■ “Assisting State efforts to build statewide systems of support for districts and 
schools in need of improvement” 

■ “Working to identify, broker, leverage, and deliver information, resources and 
services from the Content Centers and other sources” 

■ “Convening in partnership with Content Centers and others, as appropriate. States 
and districts to receive training and information on best practices and research- 
based improvement strategies” 

■ “Providing guidance and training on implementation of requirements under 
NCLB and other related Federal programs” 

■ “Facilitating collaboration at the State level to align Federal, State, and district 
school improvement programs” 

■ “Helping Content Centers to identify, document, and disseminate emerging 
promising practices” 

The Notice Inviting Applications portrayed CCs as a central source of readily accessible 
knowledge, resources, and tools. Each CC was designed to consolidate in-depth knowledge in 
one of five key content areas: Assessment and Accountability, Instruction, Teacher Quality, 
Innovation and Improvement, or High Schools. The types of knowledge specifically mentioned 
included research, scientifically valid practices, and promising practices. The degree of emphasis 
on research and scientifically valid practice was heightened in this redesign of the 
Comprehensive Centers program. This was consistent with NCLB, which stated that 
scientifically based research must inform local practice. 

Within their content areas ED tasked CCs to:^ 

■ “Identify, organize, select and translate existing key research knowledge. . .and 
communicate the information in ways that that are highly relevant and highly 
useful to State and local level policy makers and practitioners” 

■ “Benchmark State and district practices for implementing NCLB provisions and 
school improvement interventions. . .and identify promising approaches that can 
be shared with States and districts” 

■ “Convene States and districts, researchers and other experts to learn from each 
other about practical strategies for implementing NCLB provisions and programs” 

■ “Train Regional Center staff on what is known about scientifically valid practices 
and programs” 



^ Ibid., 32585-6. 
Ibid., 32586-7. 
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■ “Collaborate with Regional Centers to address specifie State requests for 
assistanee” 

■ “Communieate to the field. . .Department guidanee related to the center’s content 
focus” 

■ “Design needs assessment and data analysis tools that States and districts can use 
to benchmark their programs and progress” 



Awards to the Centers 

At the conclusion of the competition, new Regional Comprehensive Centers were located 
in 16 regions of the United States, covering all U.S. states and territories. Of the 16 RCCs, there 
were four that served only their respective state: New York, Texas, California, and Alaska. The 
remaining 12 Centers served from two to seven states and other jurisdictions. The non-state 
jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto 
Rico, the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, 
Federated States of Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, Republic of the 
Marshall Islands, and Republic of Palau. Throughout this report, the term “state” will be defined 
to include the 50 states as well as these other jurisdictions. A full list of the grantees and 
subgrantees appears in appendix A of this report. 

Comprehensive Center funding for FY 2006 totaled $56.3 million (exhibit 1.2). 

Individual Centers’ funding ranged from a low of $850,000 for the Alaska and Pacific 
Comprehensive Centers, to a high of $5,912,997 for the California Comprehensive Center. 
Funding for each RCC was driven by a formula based on the region’s total population and its 
number of poor children ages 5-17. Average funding across all RCCs was $2,876,640. 

Content Center funding for FY 2006 ranged from $1,466,096 for the Assessment and 
Accountability Center and the Center on Innovation and Improvement, to $2,446,096 for the 
Centers on Instruction, Teacher Quality, and High Schools. The latter group of Centers was co- 
funded with an allocation of $1 million each from the Special Education Technical Assistance 
and Dissemination Act, authorized under the Individuals with Disabilities Education Act. The 
average amount of funding across CCs was $2,054,096. 
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Exhibit 1.2. Comprehensive Centers funding in FY 2006 



Centers 


States 


FY 2006 
funding 


Total 




$56,296,713 


Regional Centers 






Alaska Comprehensive Center 


AK 


$850,000 


Appalachia Regional Comprehensive 
Center 


KY, NC, TN, VA, WV 


$3,829,927 


California Comprehensive Center 


CA 


$5,912,997 


Florida and Islands Regional 
Comprehensive Center 


FL, Puerto Rico, Virgin 
Islands 


$3,788,289 


Great Lakes East Comprehensive 
Center 


IN, Ml, OH 


$3,592,771 


Great Lakes West Region 
Comprehensive Center 


IL, Wl 


$3,702,196 


Mid-Atlantic Comprehensive Center 


DE, MD, NJ, PA, DC 


$3,388,147 


Mid-Continent Comprehensive Center 


AR, KS, MO, OK 


$2,111,226 


New England Comprehensive Center 


CT, MA, ME, NH, Rl, VT 


$1,644,795 


New York Comprehensive Center 


NY 


$2,886,970 


North Central Comprehensive Center 


lA, MN, ND, SD, NE 


$1,286,458 


Northwest Regional Comprehensive 
Center 


ID, MT, WY, WA, OR 


$1,630,818 


Pacific Comprehensive Center 


HI, American Samoa, 
Northern Mariana Islands, 
Federated States of 
Micronesia, Guam, Marshall 
Islands, Palau 


$850,000 


Southeast Comprehensive Center 


AL, GA, SC, LA, MS 


$4,120,988 


Southwest Comprehensive Center 


AZ, UT, CO, NV, NM 


$2,491,327 


Texas Comprehensive Center 


TX 


$3,939,324 


Content Centers 






Assessment and Accountability Comprehensive Center 


$1,466,096 


Center on Innovation and Improvement 




$1,466,096 


Center on Instruction 




$2,446,096 


National Comprehensive Center for Teacher Quality 


$2,446,096 


National High School Center 




$2,446,096 



EXHIBIT READS: The Alaska Comprehensive Center received funding in the amount of 
$850,000 for FY 2006. 

SOURCE: U.S. Department of Education 
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Background on the State Role in NCLB 



A basic premise of the Comprehensive Centers program as designed by ED was that 
NCLB assigned many tasks to states. The baekground seetion of the Notiee itemized the 
following NCLB requirements for states: 

...set standards for student performance, implement statewide testing and aecountability 
systems to measure sehool and student performanee toward aehieving those standards, 
adopt researeh-based instruetional and program improvements related to teaehing and 
learning in the elassroom, ensure that all teachers in eore subjeet areas are highly qualified, 
and improve or ultimately restructure schools that are eonsistently low-performing.^ 

With respeet to eonsistently low-performing sehools and also low-performing distriets, 
NCLB mandated that all states establish and sustain statewide systems of support and 
improvement for sehool distriets and sehools identified for improvement under NCLB. Distriets 
and sehools identified for improvement must reeeive assistanee from support teams, institutions 
of higher edueation, and regional serviee eenters in the state. This “statewide system of support” 
(SSOS) must inelude individuals who were knowledgeable about researeh and practice on 
teaching and learning and who eould develop and implement eomprehensive improvement 
strategies. “State support teams” (SSTs) were required to help sehools plan for improvement and 
to evaluate the effeetiveness of sehool personnel. The NCLB legislation also provided that these 
support teams should reeeive technieal assistanee from Comprehensive Centers and others.* 



Research Questions Addressed in This Report 

The eurrent evaluation takes a global look at the Center program as designed by ED, 
tracking the ways in whieh the Comprehensive Centers interacted with elients (both states and 
other Centers) over three program years. This report presents data deseribing implementation and 
outcomes in response to the following researeh questions: 

■ What are the objeetives of the Comprehensive Center network and of each 
Center? 

■ What kinds of produets and serviees are provided by the Comprehensive Center 
network and by eaeh Center? 

■ How do Centers develop, refine, and earry out their plans for teehnieal assistanee? 
How do they define their elients’ edueational needs and priorities? How do Center 
elients (states or Regional Centers) define their needs and priorities? 

■ To what extent is the work of eaeh Comprehensive Center of high quality, high 
relevanee and high usefulness? 



’ Ibid., 32584. 

No Child Left Behind, Title I, Part A, § 1 1 17 (a) (1). 
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■ To what extent have states relied on other sourees of teehnieal assistanee besides 
the Comprehensive Centers? What other sourees? How does the usefulness of 
Center projects compare with the usefulness of projects from other sources? 

■ To what extent do states report that Center projects have expanded state capacity 
to address underlying needs and priorities and meet the goals of NCLB? 



Questions for Future Reports 

Two additional research questions examining change over time are to be addressed in a 
final report; this report, based on data collection regarding the Centers’ work in 2007-08 and 
2008-09, will address the questions: 

■ Has the performance of the Centers in addressing the needs and priorities changed 
over the period of time studied? 

■ Has the quality, relevance, or usefulness of each Center’s projects changed over 
the period of time studied? 

The evaluation will also report on case studies of capacity building at the state level. 
These findings will provide additional insight related to the research question regarding the ways 
in which Center projects have expanded state capacity for NCLB implementation. 



Organization of This Report 

Following this introductory chapter, chapter 2 describes the study’s methods. Chapter 3 
describes the ways in which the Centers planned and conducted technical assistance for the 
2006-07 program year and it presents findings on the types of services and products they 
provided. Chapter 4 addresses the technical assistance needs and priorities reported at the state 
level and the ways in which Centers were reported to have addressed these needs and priorities, 
and expanded state capacity. Chapter 5 reports on the quality, relevance, and usefulness of 
selected Center projects as determined through expert review (for quality) and participant 
surveys (for relevance and usefulness). 
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2. Study Design 



This chapter describes the data sourees and analytie procedures used to address the 
report’s research questions. Data collection for this interim report was conducted over a period 
from May 2007 through July 2008, and addressed the activities of Centers and their clients 
during the period from July 1, 2006, through June 30, 2007. A final report will include two 
additional rounds of data colleetion in 2008-09 and 2009-10 conducted to address activities in 
subsequent years, from July 2007 through June 2009. 

The evaluation team used six data sources to address the research questions: documents 
produced by the Centers with assistance from the evaluation team. Center management plans, 
site visits to Centers, state manager surveys, expert panel reviews, and participant surveys 
(exhibit 2.1). 

For the first three questions, which pertained to the operations of the Centers, the data 
were drawn from the Centers themselves. The team used Center management plans as a data 
source regarding Center objeetives. In summer 2007 and summer 2008, the team eonducted site 
visits to Centers; closed-ended prompts were used in face-to-face interviews to gather self-report 
data systematically on Center objectives, procedures for needs assessment and planning, and 
interactions with their clients. Two doeumentary sources were used as sources for descriptions of 
Center teehnical assistance: the project inventory forms and project cover sheets completed by 
the Centers with review and feedback from the evaluation team. The definition of a project and 
the procedures for gathering and reviewing these data are discussed in detail in this chapter. 

A survey of state managers was the source for data on the states’ teehnical assistance 
needs and priorities, on other sources of teehnical assistance used, ratings of the overall teehnieal 
assistance received, and on perceived eapacity change at the state level. The technical quality of 
Center technical assistance was assessed by a panel of experts on the topic of each technical 
assistance project. Finally, participants answered survey questions pertinent to the relevance and 
usefulness of Center teehnieal assistance. 

Each of these data sources is described in this chapter, and the analytic procedures 
specific to each source are discussed. The data eolleetion instruments and further details 
regarding data sources and proeedures can be found in appendiees referenced throughout the 
chapter. The chapter concludes with a brief explanation of units of analysis for the reporting of 
quality, relevance, and usefulness ratings. 
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Exhibit 2.1. Data sources for the research questions 



Research question 


Project 

inventory 

forms 

and 

project 

cover 

sheets 


Center 

manage 

-ment 

plans 


Site 

visits to 
Centers 


State 

manager 

survey 


Expert 

review 

panels 


Partici- 

pant 

surveys 


What are the objectives of the 
Comprehensive Center 
network and of each Center? 




✓ 


✓ 








What kinds of products and 
services are provided by the 
Comprehensive Center 
network and by each Center? 


•/ 












How do Centers develop, 
refine, and carry out their 
plans for technical 
assistance? How do they 
define their clients’ 
educational needs and 
priorities? 






✓ 








How do Center clients (states 
or Regional Centers) define 
their needs and priorities? 








■/ 




•/ 


To what extent have states 
relied on other sources of 
technical assistance besides 
the Comprehensive Centers? 
What other sources? How 
does the usefulness of Center 
projects compare with the 
usefulness of projects from 
other sources? 








■/ 






To what extent do states 
report that Center projects 
have expanded state capacity 
to address underlying needs 
and priorities and meet the 
goals of NCLB? 








V 






To what extent is the work of 
each Comprehensive Center 
of high quality, high relevance 
and high usefulness? 








✓ 

(overall 

relevance 

and 

usefulness) 


V 

(quality) 


✓ 

(project- 

level 

relevance 

and 

usefulness) 
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Identification of a Sample of Center Work for Expert Review 
and Participant Surveys 



A critical component of the evaluation was to rate the quality, relevanee, and usefulness 
of Center products and services. Given available resourees, it was not possible for the evaluation 
team to submit all of a Center’s produets and serviees to an independent review panel to rate 
quality. Nor was it feasible to survey all individuals who used Center produets or partieipated in 
Center aetivities in the designated time period regarding relevanee and usefulness. Therefore, the 
evaluation team developed and applied a strategy to seleet a sample of work from each Center 
for expert panel review and partieipant ratings. 

The following sections describe: (1) the unit of analysis, (2) the sample frame, (3) the 
evaluation sample, and (4) materials obtained from Centers. 



The Unit of Analysis: The Project 

The evaluation team initially reviewed the Centers’ 2006-07 management plans to 
understand the nature of the work the Centers were eondueting and determine if the management 
plans might serve as an appropriate sampling frame for the evaluation. Based on this effort, the 
team determined that the sampling proeedures eould not be based on the management plans for 
two reasons. First, the Centers’ plans and work eontinued to evolve over time. Consequently, the 
plans, whieh were prepared before the program year, did not comprehensively refleet the work 
aetually being done by the Centers months later. Seeond, the Centers used different approaehes 
to organizing and aggregating their work. For example, some presented their work by state while 
others organized it by topie area. For the purposes of this evaluation, the team identified 
“projeets” as a eommon level of aggregation that would eonstitute units large enough for review 
and rating, but focused enough for coherence. A “projeet” was defined as a group of closely 
related activities and/or deliverables designed to achieve a specific outcome for a specific 
audience. To ensure that projeets would eonstitute units that were large enough for review and 
rating, but foeused enough for eoherenee, the study team provided the following eriteria: 



■ Complete and coherent whole. Beeause eaeh projeet should be able to stand on 
its own in an expert panel review, it should inelude all related aetivities and 
produets. 

■ Common intended outcome. Where a eluster of aetivities and deliverables was 
designed by the Center to lead to the same outeome for the same audience, those 
activities and deliverables should be grouped as one projeet. 

■ Topic area focus. With few exeeptions, a projeet addressed just one topie area 
(for example, statewide systems of support, adoleseent literacy, assessment of 
English language learners) around whieh there was a body of researeh or 
professional wisdom. 
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Since the projeet was a unit developed for this evaluation and was not necessarily how 
Centers divided up their work for programmatie or eost-traeking purposes, the evaluation team 
took several steps to ensure that the projeet eoneept was elear and used eonsistently aeross 
Centers. The evaluation team developed a standard Project Inventory Form (PIF) that Centers 
used to ereate an inventory of their work for eaeh of the program years in this evaluation (see 
appendix B for a eopy of the PIF) as well as written guidanee in the instruetions for eompleting 
the PIF (deseribed in appendix B) and a sample inventory to serve as an example of the kinds of 
projeets that should be listed by the Centers. The sample inventory ineluded examples of 
activities or resourees for defining projeets at each level of effort — major, moderate, or minor. 
Members of the study team also invited Center staff to attend training sessions by eonferenee eall 
on eompleting the PIF, ineluding defining projeets. When draft inventories were reeeived, the 
study team reviewed them to make sure the entries listed met the three eriteria used to identify 
projeets. They also reviewed the Center’s reporting of the projeet level of effort for 
eorrespondence with the listed aetivities and resourees and listing of the projeets under eaeh 
topie area for eorrespondenee with the topie definitions provided in the written guidanee. The 
study team provided technical assistance to Centers as needed to ensure that all projects 
conformed to the standards. 



The Sampling Frame 

The evaluation was designed to assess the quality, relevanee, and usefulness of a sample 
of Center work. The PIFs served as the basis for identifying the sampling frame from whieh the 
study team drew the sample of projeets that beeame the subjeet of expert panel reviews and 
partieipant surveys. For the 2006-07 data eolleetion eyele, the evaluation team asked each 
Comprehensive Center to use the PIF to prepare an inventory of all the projeets aetive during the 
grant period from July 1, 2006 through June 30, 2007. A total of 364 projects (110 major, 106 
moderate, and 148 minor) were identified aeross the 21 PIFs. The number of projeets ineluded 
on the 2006-07 PIFs ranged from a minimum of 6 to a maximum of 32 projeets. 

To be eligible for the sampling frame, projects first needed to represent a reasonable 
amount of effort (i.e., classified as “major” or “moderate” by the Center) and have a suffieient 
amount of material to give reviewers enough information to judge the quality of the work. 
Projeets ineluded in the sampling frame also needed to have identifiable partieipants sinee the 
evaluation design ealled for eollecting relevanee and usefulness ratings through surveys of 
project participants. All 148 minor projects were exeluded from the sampling frame, along with 
16 major or moderate projeets that had produeed few or no reviewable materials or did not have 
identifiable partieipants in 2006-07. A total of 200 projeets met both eligibility eriteria and were 
ineluded in the sampling frame. 



Centers designated on their PIFs whether each project was “major,” “moderate,” or “minor” in terms of the level 
of effort and/or resources the Center devoted to it, relative to other projects in the same Center. 
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The Sample of Projects 

To ensure that the final sample of projects reviewed reflected a range of each Center’s 
work across topics and states and captured work that represented the largest investment of 
resources, the team implemented a sampling strategy that included a combination of Center- 
nominated projects and a stratified set of purposive ly-selected projects. The desired sample size 
at each Center was a function of that Center’s budget amount. For 2006-07, the number of 
projects sampled for each Center ranged from three to eight depending on the size of the Center’s 
annual budget (exhibit 2.2). Although the sample was not statistically representative in scientific 
terms, it was designed to include a high percentage of the major projects of each Center as well 
as projects that Centers thought best represented their work. 



Exhibit 2.2. Sample size by budget 



Center funding level — 
annual federal contract 
(number of Centers) 


Total 

projects to 
be selected 
for review, 
per Center 


Number of 
projects to be 
nominated by 
Centers, per 
Center 


Number of 
projects to 
be selected 
purposively, 
per Center 


Number of 
projects 
selected 
across 
Centers 


Total 








122 


Less than $1 million 
(N=2) 


4 


1 


3 


Y** 


$1 to 1.9 million 
(N=5) 


5 


1 


4 


24 


$2 to 2.9 million 
(N=6) 


6 


2 


4 


35** 


$3 to 3.9 million 
(N=6) 


7 


2 


5 


40** 


$4 million or more 
(N=2) 


8 


2 


6 


16 



* Does not include a collaborative project conducted jointly by an RCC and a CC that is accounted for in 
another category. 

** In four cases, Centers did not have enough eligible projects in their sampling frame to meet the desired 
sample size. 

EXHIBIT READS: There were four Centers whose annual funding level was less than $1 million. Three 
projects were selected for review for each of these centers, one of which was nominated by the Center 
for selection and two of which were selected for inclusion by the study team. 



When completing their inventory forms. Centers were given an opportunity to nominate 

2 1 

the one or two projects they felt best represented their work. These Center-nominated projects 



The number of projects a Center was able to nominate depended on the size of its annual budget, as shown in 
exhibit 2.2. 



13 








were seleeted first for the sample, followed by major projeets. For Centers with in s ufficient 
major projects to meet the predetermined sample size, moderate projects were selected. The 
study team used an iterative sampling process to randomly select projects while controlling for 
topic area and state. 

Using these methods for 2006-07, the evaluation team selected a total sample of 122 
projects*-93 major and 29 moderate (see exhibit 2.3). Overall, 61 percent of the projects that 
were eligible for the sampling frame (122 of 200) were selected. The sample predominantly 
included the most major work of the Centers (90 percent of the major projects) although the 
sample was not statistically representative in scientific terms, and covered 34 percent of all 
projects reported on the PIFs. 

Exhibit 2.3. Project sampling process 



Sample frame 
(N=200) 



Study sample 
(n=122) 




EXHIBIT READS: Of the 364 projects listed on the PIFs, 164 were excluded from the sample 
frame, leaving 200 eligible projects from which 122 were sampled. 



Further detail on the sampled projects by topic in relation to all major and moderate 
Center projects appears in exhibit 2.4. For 17 of the 23^ topics, 50 percent or more of all major 
and moderate projects addressing that topic were sampled. 



* This total includes one collaborative project conducted by an RCC and a CC together that was classified as a major 
project by both Centers. At the end of this chapter, in the section detailing the study’s analytic procedures, the 
handling of this project in analysis and reporting is described. 

^ Exhibit 2.4 includes the topic area of “other” as the 23“^ topic area, as a small number of the sampled projects were 
included in that area. Subsequent discussions in the report use the 22 specific substantive topic areas. 
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Exhibit 2.4. Distribution of aii major and moderate projects and projects 
in the evaiuation sampie, by topic 



Project topic 


Number of 
major and 
moderate 
projects on 
project 
inventories 


Number of 
projects in the 
sample 
(all major or 
moderate) 


Percent of all 
major and 
moderate 
projects in 
evaluation 
sample 


Total 


216 


122 


56% 


Components of effective systems of 
support — state, district, schooi 


54 


36 


67 


Data use/data-driven decisionmaking 


5 


2 


40 


Formative assessment 


5 


1 


20 


Reading 


17 


7 


41 


Adoiescent iiteracy 


11 


5 


45 


Mathematics 


11 


6 


55 


Dropout prevention 


4 


2 


50 


High schooi redesign/reform 


10 


5 


50 


Transition to high schooi 


4 


3 


75 


Speciai education — curricuium, instruction, 
and professionai deveiopment 


5 


3 


60 


Speciai education — assessment 


1 


1 


100 


Engiish ianguage iearners 


17 


11 


65 


Highiy quaiified teacher provisions of NCLB 


8 


5 


63 


Teacher preparation and induction 


6 


4 


67 


Teacher professionai deveiopment 


8 


7 


88 


Suppiementai educationai services (SES) 


9 


5 


56 


Response to Intervention (Rtl) 


7 


3 


43 


Migrant education 


4 


2 


50 


Indian/native American education 


2 


1 


50 


Data management compiiance 


8 


5 


63 


Assessment design 


2 


2 


100 


Parent invoivement 


6 


3 


50 


Other 


12 


3 


25 



EXHIBIT READS: There were 54 major and moderate projects on the project inventories that focused on 
components of effective systems of support. Of these, 36 (or 67 percent) were in the project sample. 

SOURCE: Project inventory forms submitted by the Centers. 
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Materials from Centers 



The evaluation team notified the 21 Center Direetors about projects selected for review 
and sent a standard Request for Materials for Expert Panel Review (see appendix B for a copy of 
the transmittal memo and request form). Centers were asked to assemble and submit a 
comprehensive set of pre-existing materials associated with each project (meeting agendas, 
briefing books, meeting summaries, training materials, white papers, web resources, etc.) that 
would fully describe the project and provide reviewers with a sufficient basis for rating the 
technical quality of the work. Centers were also asked to include a participant list and a standard 
cover sheet, using a format developed by the evaluation team, for each project. The participant 
lists were used to draw the sample for participant surveys. 

When the project materials were received, they were reviewed by evaluation team 
members for completeness. If materials were missing or inaccessible (e.g., electronic files didn’t 
open) or the cover sheet was incomplete, a follow-up memo was sent to the Center detailing the 
issued identified. Evaluation team members then worked with the Center to obtain the missing 
information and finalize the review package (see appendix B for a copy of the follow-up memo). 



Describing Center Operations 

The purpose of collecting data from the Centers was to describe Center operations and to 
address research questions regarding the objectives of the Centers, the kinds of products and 
services provided, how the Centers defined their clients’ needs and priorities, and the extent to 
which the Centers met the objectives of their own plans. The description of Center operations in 
chapter 3 of this report is drawn from three sources of data: interview responses gathered in site 
visits to the Centers, PIFs, and the project cover sheets submitted for the sampled projects. The 
procedures used in gathering and analyzing data from these sources are described in the 
following pages. 



Management Plans 

Centers stated their objectives for the year in their annual program management plans. 
For the 2006-07 program year. Centers had developed their management plans in spring 2006, 
prior to the program year. The evaluation team relied on these management plans to gather 
information from statements of the Center’s intended substantive focus for the year’s technical 
assistance. The team coded each Center’s stated objectives in their management plan by subject 
area, using the same list of 22 topic areas used to code the PIFs. Appendix C provides a 
description of the process used to code the management plans as well as an analysis of the 
intercoder reliability. 
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Site Visits to Centers 



Site visits were eondueted at two time points, first in summer 2007, followed by a visit in 
summer 2008 to capture descriptions of Center operations during the 2006-07 program year. As 
described earlier, the team reviewed Center management plans in 2006-07 to understand the 
nature of the work the Centers were conducting; this review provided a way of preliminarily 
identifying objectives, approaches, and procedures found in Center work. Structured site visit 
interviews were conducted with Center management teams in summer 2007, posing open-ended 
questions regarding the following topics (see appendix C for the protocols): 

■ Center organization (lead organization, subgrantees, ways of dividing 
responsibilities among staff) 

■ Major areas of focus 

■ Communication with client organizations (states in the case of RCCs, or RCCs in 
the case of CCs) regarding needs and assistance to be provided 

■ Modes of delivering technical assistance 

■ Sources of knowledge used 

■ Approaches taken in quality assurance 

■ Working relationships within the Center network (between RCCs and CCs, with 
other Centers of the same type, and networkwide) 

■ Barriers or challenges encountered 

The team’s review of the 2007 interview transcripts informed the development of closed- 
ended prompts that were subsequently administered to all Center leaders in interviews in the 
following summer. These second-round structured interviews included binary questions to 
follow up about Center objectives for 2006-07 and activities that had occurred during the 
program year (see appendix C for the interview prompts). It is the responses to these prompts 
that are reported, across Centers, in chapter 3. 



Project Inventory Forms (PIFs) 

While the main purpose of the PIFs was to build the sample frame (described earlier in 
this chapter), the evaluation team also used the PIFs to gain Center-level data on the work the 
Centers undertook in 2006-07. The projects sampled for quality, relevance, and usefulness 
ratings were classified by topic area (see appendix C). The evaluation team coded the topics of 
the remaining projects using the same 22 topic areas to provide a full picture of the topics 
covered by each Center’s 2006-07 PIF. Appendix C provides a description of the process used to 
code the projects as well as an analysis of the intercoder reliability. 
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Project Cover Sheets 



For projects included in the sample, the standard cover sheets provided by the Centers 
described project activities and cross-Center collaboration in the project. The project cover 
sheets were primarily collected from each Center to help orient expert panels to the purpose and 
content of the materials to be reviewed (see appendix B for a copy of the Request for Materials 
for Expert Panel Review transmittal memo and cover sheet). 

The evaluation team used the cover sheets as a data source for an overall description of 
activities and collaboration in the sampled projects (reported in chapter 3) and for use in 
classifying projects into subgroups for which the ratings of relevance and usefulness could be 
analyzed (reported in chapter 5, as described below). The cover sheets provided descriptive 
information for each project, including the activities and deliverables associated with the project, 
and the contributions of other Centers to the project. Categories of Center activities and 
resources were drawn from review of the Center management plans and site visit interviews. 
These coding categories, thus, permitted a yes/no judgment of whether each project offered each 
of the following activities or resources to participants: 

■ Ongoing consultation and follow-up 

■ Research collections or syntheses 

■ Engagement of participants in project planning 

■ Training events 

■ Task force meetings and work 

■ Conferences 

■ Support development of a formal plan to implement a program or policy 

The team also coded the type of contribution, if any, of any CC to each RCC project, and 
of any RCC to each CC project. All these elements of the cover sheets were coded by members 
of the evaluation team using procedures described in appendix C; the appendix also provides 
results of the analysis of intercoder reliability. 



Survey of State Managers 

The purposes of the survey of state managers were to obtain information on state 
priorities in terms of state responsibilities related to the implementation of NCLB, obtain the 
state perspective on the relevance and usefulness of Center assistance, and obtain a comparative 
judgment of Center assistance in relation to assistance available through other sources such as 
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professional associations. The survey instrument used for the survey of state managers appears in 
appendix D of this report. 

In order to identify appropriate respondents for the survey of state managers, the 
evaluation team collected the names of each RCC’s main points of contact in each SEA during 
the summer 2007 site visits to the RCCs. For 2006-07, the resulting list of state managers 
included a total of 132 respondents across the 62 states included in the study. 

To be fully reflective of all SEAs in the analysis of these data, it was critical that the 
study team receive completed surveys from state managers in each state. The final response rate 
for the state manager survey was 100 percent for the 50 states and 50 percent for the 12 outlying 
areas, for an overall response rate of 90 percent. 

For 20 states, the evaluation team received completed responses from more than one state 
manager (exhibit 2.5). The state was the primary unit of analysis in analyzing data from the state 
manager survey for this report. The state managers’ responses were weighted to ensure that each 
state was equally represented in all summary statistics while taking into account the variation in 
responses within each state. The weighting procedure, where each response was weighted by the 
inverse of the number of managers responding from that state, ensured that each state was 
equally represented when the evaluation team aggregated responses across states to describe the 
distribution of responses. 



Exhibit 2.5. Number of responses to the state manager survey 



Number of responses 
from the states 


Number of 
states 


Total 


56 


1 


36 


2 


15 


3 


4 


4 


1 



EXHIBIT READS: In 36 states, the survey of state managers was 
completed by a single respondent. There was one state where the state 
manager survey was completed by four separate respondents. 



Across states, the offices, divisions, or departments represented by state managers were 
federal programs (in 68 percent of states), school improvement (64 percent), curriculum and 
instruction (53 percent), assessment and accountability (40 percent), and special education (30 



24 

The Comprehensive Centers were expected to serve 62 jurisdictions including the 50 states and 12 other 
jurisdictions: the District of Columbia, Puerto Rico, the Virgin Islands, American Samoa, Commonwealth of 
Northern Marianas, Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the 
Marshall Islands, and Republic of Palau. For purposes of this report, the term “state” refers to the 50 states and the 
outlying territories listed here. 
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percent). In relation to managing technical assistance, state managers were most frequently 
responsible for identifying needs and priorities (84 percent of states) and managing others in 
their use of technical assistance services (80 percent). Other frequent job responsibilities 
included serving as point of contact or manager for specific technical assistance projects (66 
percent), seeking out technical assistance providers (65 percent), and negotiating a scope of work 
with technical assistance providers (59 percent). 



Assessment of Quality by Expert Panels 

Given the Centers’ charge to focus on scientifically based research and evidence-based 
practices to improve student achievement and close achievement gaps, one goal of the 
Comprehensive Centers evaluation is to assess the technical quality of work across the Center 
network using an independent panel of expert reviewers. Each sampled project was 
independently rated by a panel of three experts. Reviewers assigned a score to each of three 
quality dimensions discussed below, using a 5-point rating scale. 

In an effort to maximize interrater reliability, the evaluation team: (I) defined quality and 
developed a detailed scoring rubric that could be applied across all Centers and a range of 
projects; (2) recruited, trained and assigned highly qualified expert panelists; and 
(3) implemented a process for reviewers to discuss their findings with one another when scores 
were discrepant for particular projects. The following sections describe each of these steps, 
followed by a brief discussion of how the final technical quality ratings were calculated. 



Define Technical Quality and Develop Scoring Rubric 

Developing the definition of technical quality and the rubric used to measure it 
incorporated information from many sources, including federal legislation and ED specifications 
on what constituted scientifically based evidence and an example from another federal agency. 
When defining technical quality and developing the scoring rubric, the evaluation team sought to 
ensure that the definition was relevant to the range of projects Centers would provide- 
recognizing that some projects might have a substantial research knowledge base and others 
might be guided more by promising practices (those that were supported by evidence but not yet 
rigorously studied) or legislative or regulatory requirements. Also, the definition of quality and 
the associated rubric had to be applicable to projects at varying points of development and 
implementation, from early-stage needs assessment and design work to fully-developed products 
and services. The evaluation team developed a quality scoring rubric, included in appendix E, to 
assess quality along the following three dimensions: 

■ Dimension I : Demonstrated use of the appropriate documented knowledge base 



Two panelists were not able to complete their reviews in a timely fashion, resulting in nine projects having been 
rated by only two panelists. 

Review of Instructional Materials for Middle School Science. (1997, February). National Science Foundation, 
Retrieved from http://www.nsf gov/pubs/1997/nsf9754/nsf9754.htm?org=NSF . 
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■ Dimension 2: Fidelity of applieation of the knowledge base to the products and 
services provided 

■ Dimension 3: Clear and effective delivery 

Reviewers assigned a score to each dimension, using a 5 -point rating scale (where 1 
meant “very low quality” and 5 meant “very high quality”), according to the indicators defined 
for each dimension and examples in the scoring booklet. 



Recruit, Train and Assign Expert Paneiists 

To meet the selection criteria for this evaluation, expert panelists had to have current, 
rigorous work in the particular topic of interest (for example, publications in peer-reviewed, 
scholarly journals; presentations at relevant professional organization meetings; recent 
membership on advisory panels or task forces) and be free of conflicts of interest. Nominations 
for panelists were made by staff in ED’s Institute of Education Sciences (lES), the Technical 
Work Group for this evaluation, members of the evaluation team, and Center staff Based on the 
selection criteria, the evaluation team selected a total of 70 expert panelists, 94 percent with a 
doctorate degree and two-thirds (67 percent) with university affiliations, to review the Centers’ 
2006-07 sampled projects. 

The evaluation team sought to maximize interrater agreement in scoring of quality by 
training expert reviewers to systematically use a standard rubric. During the two-day training, 
experts worked in small groups to discuss how each of the three quality dimensions, and their 
corresponding indicators, applied to sample project descriptions that were provided by the study 
team. Reviewers independently scored one of two projects overnight and submitted their 
dimension-level scores at the beginning of the second day. Scores were posted and analyzed in 
terms of rater agreement at the dimension level. The panelists again worked in small groups to 
discuss their scores and identify possible reasons for any discrepancies in the results. During the 
small group discussions, evaluation staff circulated among the groups to assess whether 
reviewers had adhered to the standards of evidence discussed on Day 1 

During the actual review process, expert reviewers were asked to score four to eight 
projects in their area(s) of expertise, with no more than three projects from any given Center. 
Since judgments about the state of the available evidence on a given topic and its applicability to 



Current Comprehensive Center staff, as well as individuals employed by organizations that had an ongoing 
financial relationship (for example, a contract or cooperative agreement) with a Comprehensive Center and who 
worked on a Center project, were not eligible to serve as reviewers. 

The final expert panels were made up of 67 expert reviewers. One of the initial trainees was removed due to a 
conflict of interest, and two reviewers were later dropped from the review process when they failed to complete their 
reviews in a timely fashion. 

During training, expert reviewers scored a sample project and discussed their scores in small groups, paying 
particular attention to areas where scores were divergent. This approach allowed the evaluation team to identify 
dimensions or indicators within the scoring rubric that seemed to be problematic (resulting in discrepant scores) or 
particular reviewers who needed additional training or appeared to be inappropriate to use. The goal was to have 
panelists leave the training with a common understanding of how to apply the quality rubric; the evaluation team did 
not intend to attempt to establish a specific interrater reliability criterion at training. 
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the project being rated relied heavily upon the knowledge of the expert reviewers, it was 
important that reviewers were assigned projects that matched their area(s) of expertise. The 

30 

evaluation team was also careful when assigning projects to avoid known conflicts of interest. 
When needed, the evaluation team reassigned projects when reviewers were unable to complete 
their reviews in a timely fashion, identified unforeseen conflicts of interest, or did not feel they 
had the requisite expertise to review the assigned projects. 

Overall, 1 13 of the 122 projects sampled were reviewed by three panelists. The 
remaining nine projects were reviewed by only two panelists because the third assigned reviewer 

3 1 

did not complete the review within a reasonable timeframe. 



Address Interrater Reliability 

In addition to the detailed scoring rubric and training provided, a resolution process was 
used to help achieve a high degree of interrater reliability in scoring. If the reviewers’ project- 
level scores (defined as the simple average of their three dimension-level scores) were found to 
differ by 2 or more points from each other for any given project, the evaluation team convened 
the panel by telephone to discuss the ratings. The goal of the discrepancy conference calls was 
to give panelists an opportunity to understand the rationale behind their colleagues’ scores and 
consider whether, on the basis of that discussion, any scoring revisions were warranted. The 
evaluation team emphasized to the reviewers that these discussions were not intended to achieve 
consensus among them. 

Overall, 24 percent (29 of 122) of the projects had discrepancies in the project-level 
scores (exhibit 2.6). In 7 of the 29 discrepant cases, there were discrepancies in two of the three 
dimensions, while the remaining 22 projects had discrepancies in all three dimensions. After a 
discrepancy conference call, panelists were given the opportunity to submit a revised scoring 
form and narrative report of the project’s strengths and weaknesses. Although panels were not 
required to reach consensus, the discrepancy was resolved in all but two cases, increasing the 
number of projects in agreement from 76 percent to 98 percent. 

To estimate interrater agreement, we calculated the rWG(J) index. Before the 
discrepancy calls, the average rWG(J) coefficient was 0.75.^"^ After the calls were conducted, the 
value increased to 0.87. 



For the purpose of assignments, the term “confliet of interesf ’ meant any finaneial or other interest that appeared 
to conflict with or significantly compromise the service of the individual reviewer because it could significantly 
impair the individual’s objectivity. 

In consultation with IBS, the evaluation team decided to forgo the third review on these particular projects since 
there was interrater agreement (i.e., a difference less than 2 points) between the two scores that had been submitted 
by the other panelists assigned to these projects. 

The 2-point difference threshold was selected in an effort to identify projects with widely divergent scores that cut 
across qualitative categories (i.e., “low” quality versus “high” quality), while at the same time minimizing burden on 

expert panelists. 

33 

James, L.R, Demaree R.G., and Wolf G. (1984). Estimating within-group interrater reliability with and without 
response bias. Journal of Applied Psychology, 69(1), 85-98. 
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Exhibit 2.6. Distribution of discrepancies and results of the discrepancy 
call process 



Description 


N 

(percent) 


Total number of projects reviewed 


122 


Number of projects without a project- 
level discrepancy 


93 

(76 percent of 122) 


Number of project-level discrepancies 
identified 


29 

(24 percent of 122) 


Number of projects with 
discrepancies in 2 dimensions 


7 

(24 percent of 29) 


Number of projects with 
discrepancies in 3 dimensions 


22 

(76 percent of 29) 


Number of project-level discrepancies 
resolved* by panel calls 


27 

(93 of 29 calls conducted) 


Number of project-level discrepancies 
remaining after panel calls 


2 

(7 of 29 calls conducted) 



* Discrepancies were considered resolved when the project-level scores differed by less than 2 points. 

EXHIBIT READS: Of the 122 projects reviewed, 93 (76 percent) had no discrepancies in the project level 
scores. 



Calculate Aggregate and Dimension-Level Measures of Technical Quality 

In order to analyze the teehnieal quality of sampled projeets, a series of steps was taken 
to eombine individual expert ratings into aggregate scores. The first step was to combine the 
individual dimension-level scores into a single rating from each panelist. For each project 
reviewed, a simple average of the three dimension-level scores was computed to generate a 
project-level score for each reviewer. In the second step, the three reviewers’ scores were 
averaged to determine the overall quality score for each project. In the third step, the overall 
project scores across the set of sample projects were averaged within each Center to calculate a 
Center-level quality score. 



34 

We estimated interrater agreement using the rWG(J) index. Treating each dimension as an “item,” we first 
calculated the score variance between raters on each dimension and then averaged the three variances. We assumed 
the random error variance to be 2.0 for a 5-point scale, as suggested by James et al. (1984). An rWG value between 
0.71 and 0.90 is generally considered an indicator of “strong agreemenf’ (LeBreton, J.M., and Senter, J.L. (2008, 
October). Answers to 20 questions about interrelated reliability and interrater agreement. Organizational Research 
Methods, 11, 815-852. Retrieved from http://orm.sagepub.eom/cgi/content/abstract/l 1/4/815) . 
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Center-level seores were aggregated to ealeulate the average quality rating aeross 
sampled projeets for the Center program as a whole, as well as for two subgroups of interest, the 
16 RCCs and the 5 CCs. Eaeh Center was given equal weight in eomputing the overall 
programwide rating for teehnieal quality, as well as the mean ratings for the RCCs and the CCs 
respeetively, for the sampled projeets; within Centers, eaeh projeet was given equal weight. 



Survey of Project Participants 

The purpose of the partieipant survey was to obtain client views of technical assistance 
from the Comprehensive Centers, particularly in the areas of relevance and usefulness. As the 
primary role of the RCCs is to provide technical assistance to the states and territories in their 
region, the clients for the RCCs included state-level staff. The clients for the CCs included both 
RCC staff and state-level staff corresponding with the primary role of the CCs to build the 
capacity of the RCCs as well as to work with RCCs in providing technical assistance to states. 

The evaluation team developed two parallel survey forms for project participants: one for 
state-level staff who participated in any Center project, and one for RCC staff who participated 
in a CC project. In sampling participants to respond to surveys focused on specific projects, the 
goal was to identify a sizable number of participants, drawn from complete lists of all 
participants, so that their responses would collectively provide a picture of all participants’ views 
regarding the sampled projects. The evaluation team drew samples of participants in the 122 
projects that were selected for expert panel review. In this way, expert panel ratings of quality 
and participant ratings of relevance and usefulness were gathered for the same set of projects. 

The survey instruments used for state-level and RCC staff are located in appendix E of this 

-5 c 

report. A brief description of the process used to calculate the measure of relevance and 
usefulness is provided at the end of this section. 

Centers were asked to furnish full lists of all participants and their contact information for 
each sampled project. These lists included individuals who had participated in projects in 
numerous ways including: (1) serving on task forces, school support teams, and work groups 
associated with the project; (2) attending conferences, technical assistance retreats, and other 
meetings held as a part of the project; or (3) receiving written materials or other disseminated 
resources. State-level participants included staff who were employed by SEAs as well as 
employees of intermediate agencies, EEAs, schools, or other agencies who had responsibilities 
for state-level implementation of NCEB and were participants in both RCC and CC projects. 
RCC staff, as clients of the CCs, were participants solely in CC projects. A total of 3,904 
participants who met the participation criteria were eligible for the sampling frame. 

After identifying the sampling frame, the team implemented a sampling strategy that 
combined random sampling from each project and a replacement strategy to minimize 



Since the survey asked respondents for their experiences with the Comprehensive Center system in relation to a 
specific project, the survey team provided each respondent with a list of the activities included in that project, based 
on the content of the project cover sheets provided by the Centers. For the on-line survey, this information was 
displayed on the first screen of the survey. For the paper version of the survey, the project-specific information was 
printed on yellow paper and inserted into the front of the booklet. 
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respondent burden. A sampling strategy was developed for selecting a sufficient number of 
participants to obtain fair representation of client views for each project, and across projects, 
while balancing respondent burden and data collection costs. The team drew a simple random 
sample of participants within each sampled project using the following sampling rules, based on 
the number of participants in the project: 

• All participants in projects with 12 or fewer participants were sampled. 

• A random sample of 12 participants was selected for projects with 13 to 25 
participants. 

• A range of 12 to 48 participants were randomly selected to represent 48 percent of 
participants for projects with 26 to 100 participants. 

• For each project with more than 100 participants, a random sample of 48 participants 
was selected. 

After drawing a random sample for each project, replacements were made for 
respondents who had been identified as a participant across multiple projects. To help reduce 
respondent burden, 13 percent of state-level staff and 14 percent of the RCC staff were replaced 
by alternates. These replacements resulted in 65 percent of the sample being requested to 
complete a single survey and 35 percent requested to complete more than one survey. 

Exhibit 2.7 provides an overview of the participant survey sampling and administration. 
Of the 3,904 participants in the sampling frame, the resulting sample for the participant survey 
consisted of 1,658 participants across the 122 projects. 

The evaluation team initially administered the participant survey to the sampled 
respondents online. Nonrespondents were contacted by phone, sent reminder cards, and sent 
paper survey forms, all in an effort to obtain completed responses. The final response rate for the 
participant survey was 73 percent across the 122 sampled projects. For two of these projects, no 
usable survey responses were received. Nonrespondents included participants who did not return 
a completed survey because they no longer worked for the state organization or RCC, as well as 
participants whose contact information was incorrect. 
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Exhibit 2.7. Survey of project participants sampling and survey administration 
summary 



Sample frame 
(N=3,904) 



Study sample 
(n=l,658) — 



Survey 

administration 




EXHIBIT READS: The sample frame of 3,904 project participants for survey administration included 2,689 
participants in RCC projects and 1,215 participants in CC projects. 



Calculate Measures of Relevance and Usefulness 

For the relevance and usefulness questions in the participant survey, respondents were 
asked to rate each aspect of relevance and usefulness (exhibit 2.8) using a 5 -point scale ranging 
from 5 (a very high degree) to 1 (a very low degree). 

A preliminary step in the analysis of responses was to assess the properties of the items 
included in the surveys for constructing indices of relevance and usefulness. Construction of 
these indices was based on item response from the 1,208 completed surveys. Principal 
components analysis with no rotation was conducted on each set of items in the relevance and 
usefulness scales to determine the underlying dimensions represented by these items. Among the 
items comprising the relevance scale, one factor was extracted with an eigenvalue of 5.4 and 
explained 67.6 percent of the variance (see exhibit 2.8 for factor loadings). Similarly, among the 
usefulness items, one factor was extracted with an eigenvalue of 7.6 and explained 69.2 percent 
of the variance (see exhibit 2.8 for factor loadings). The reliability of each of the scales was 



The response eategory, “Not able to judge,” on the survey instruments indieated those respondents who were not 
appropriate for addressing the partieular item and were, therefore, not ineluded in analyses. None of the respondents 
had indieated “not able to judge” for the eight items of the relevanee seale. A range of 3 to 4 pereent of respondents 
aeross the 1 1 items of the usefulness seale had indieated “not able to judge.” 
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evaluated using Cronbaeh’s Alpha. For the eight items in the relevanee seale, the reliability 
eoeffieient is 0.93, and for usefulness the eoeffieient is 0.95 aeross the 1 1 items. 



Exhibit 2.8. Reievance and usefuiness items from the project participant surveys 



Loading 
on factor 

Relevance 1 

Based on your experience, to what degree 



was this set of activities and resources 
relevant to your work, in each of the 
following respects? 

a. Addressed a need or problem that my 0.83 
organization faces 

b. Addressed an important priority of my 0.82 
organization 

c. Addressed a challenge that my 0.81 

organization faces related to the 
implementation of NCLB 

d. Provided information, advice, and/or 0.86 
resources that could be directly 

applied to my organization’s work 

e. Addressed our particular state context 0.80 

f. Addressed my organization’s specific 0.83 

challenges (e.g., policy environment, 
leadership capacity, budget 

pressures, local politics) 

g. Provided information, advice, and/or 0.85 
resources that could be used to guide 
decisions about policies, programs, or 
practices 

h. Highlighted the implications of 0.78 

research findings (or information 

about best practice) for policies, 
programs, or practices 



Loading 
on factor 



Usefulness 1 



Based on your experience, to what degree 
was this set of activities and resources 
useful to you, in each of the following 
respects? 




a. 


Provided resources that were easy to 
understand and easy to use 


0.82 


b. 


Employed an appropriate format (e.g., 
a work group, a conference, individual 
consultation, written products) 


0.81 


c. 


Provided adequate opportunity to learn 
from colleagues in other states 


0.64 


d. 


Included adequate follow-up to support 
the use of new information and 


0.82 




resources 




e. 


Were timely 


0.84 


f. 


Helped my organization solve a 
problem 


0.87 


g- 


Helped my organization maintain or 
change a policy or practice 


0.83 


h. 


Helped my organization take the next 
step in a longer-term improvement 
effort 


0.85 


i. 


Provided my organization with 
information or resources that we will 


0.88 




use again 




j- 


Helped my organization develop a 
shared expertise or knowledge base 


0.88 


k. 


Helped individuals in my organization 
to develop skills that they will use again 


0.87 



Mean ratings at the respondent level were averaged so that each respondent for a given 
project contributed equally to a project-level rating. Thus, the relevance or usefulness rating at 
the project level was a mean of the ratings provided by sampled participants in that project 
(ranging in number from 1 to 48) who returned surveys. Next, the rating of each sampled 
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Across the eight items on the relevance scale, the rate of item nonresponse ranged from 0.3 percent to 1.1 percent. 
The rate of missing data among the 1 1 usefulness scale items was slightly higher with the percent of item-level 
nonresponse ranging from 1.1 to 2.0 percent. For both relevance and usefulness calculations, respondents with 
missing data on more than two of the items were excluded from the calculation, resulting in a small number of 
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project contributed equally to the computation of the mean rating across projects for each Center. 
Finally, the team calculated an overall mean across the 21 Centers and overall means for the 16 
RCCs and the 5 CCs, respectively, for the relevance and usefulness ratings of the sampled 
projects. In calculating these overall means, each Center’s mean rating was weighted equally.^^ 



Units of Analysis for Quality, Relevance, and Usefulness 
Data in this Report 

The quality, relevance, and usefulness ratings of the sampled projects were used in 
analyzing and reporting data for several different units of analysis: the projects sampled from the 
program as a whole (all 21 Centers), the projects sampled for the 16 RCCs and the 5 CCs 
respectively, subgroups of the 122 projects, and (for relevance and usefulness) subgroups of the 
3,652 sample-eligible participants. For a Center’s mean rating on quality, relevance, or 
usefulness, the team weighted each sampled project from that Center equally; for a mean across 
the entire program or a set of Centers, the team weighted each Center’s mean equally. 

Several analyses were conducted for subgroups of projects across Centers — e.g., those 
projects that included a research synthesis among their products, or included training among their 
services. Each rater (an expert or survey respondent) had equal weight in the individual project’s 
rating, and each project had equal weight in the subgroup mean rating. 

Finally, some analyses were conducted for subgroups of participants across projects. For 
example, the ratings of relevance provided by participants who spent at least three days in a 
project activity were compared with the ratings provided by participants who spent less time. In 
these analyses, the unit was the participant, not the project, and the weighting was designed to 
permit generalization to all the sample-eligible participants across all the sampled projects. Thus 
the weight for each participant’s response was the inverse proportion of the number of 
respondents compared with the total number of participants in the sample-eligible population for 
that project. 



Number of Projects Varying with the Unit of Anaiysis 

Readers will notice slight variations in the total number of projects shown in the exhibits 
and analyses throughout this report. These figures differ for two reasons, which are explained in 
detail here (exhibit 2.9). 

First, there were two RCC projects for which the evaluation team did not receive any 
completed surveys. This meant that for any analysis incorporating participant data, the maximum 



respondents being exeluded (6 respondents [0.5 percent] for relevance and 1 8 respondents [1.6 percent] for 
usefulness). 

The evaluation team reviewed the differences in relevance and usefulness ratings between RCC staff and state- 
level staff (in ratings of those CC projects that served both types of staff) to address concerns of potential bias. No 
statistically significant differences were found in the relevance and usefulness ratings provided by RCC and state- 
level staff (see additional details presented in chapter 5, exhibit 5.17). 
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number of projects in the analysis was 94 RCCs projects and 120 total projects. For analyses that 
did not rely on participant data (e.g., the quality ratings), the team used 96 RCC projects and 122 
total projects. 

Second, as described earlier in this chapter, one project included in the sample was 
sponsored jointly by an RCC and a CC. This “combined project” was one of the projects counted 
in analyses specific to RCC projects, and also one of the projects counted in analyses specific to 
CC projects. For analyses across all projects or all participants, however, the joint RCC/CC 
project or its participants were counted only once in the analysis. 



Exhibit 2.9. Project sample, by type of rating and unit of analysis 



Original sample: 
122 projects 



RCC CC 



-> projects 


projects 




(n=95) 


(n=26) 




' 


' ’ 


r ^ r 



Combined 
RCC/CC project 
(n=l) 





NOTE: Of the 95 RCC projects, relevance and usefulness ratings were not received from participants 
for two RCC projects. Thus, only 93 RCC projects were included in the analyses for the relevance and 
usefulness ratings. 

EXHIBIT READS: Among the 122 Comprehensive Center projects in the sample, 95 were RCC 
projects, 26 were CC projects, and one project was sponsored jointly by a RCC and a CC. 
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3. How the Centers Operate 



Understanding how Comprehensive Centers operate is an important part of assessing 
their performance. As described in chapter 1, ED established certain structures and expectations 
for the effective functioning of the Centers, including the split of responsibilities between the 
two types of Centers, the emphasis on using and applying scientifically based research to build 
state capacity to carry out NCLB, and the flow of communication between and among the 
Regional Comprehensive Centers (RCCs), the Content Centers (CCs), and state agencies. 

This chapter examines how the Comprehensive Centers reported working with their 
client organizations to assess technical assistance needs and to formulate their annual work. It 
provides information about the range of technical assistance topic areas, products, and services 
delivered by the Centers and the processes by which these activities were determined. The 
analysis draws on information gathered from the Centers’ submitted Project Inventory Forms 
(PIFs), Center annual management plans, and interviews with Center staff conducted by the 
evaluation team.^^ 

The key findings include: 

■ The Centers reported planning their work in coordination with their clients. All 
16 RCCs reported obtaining state input into the initial plans and engaging states in 
refinements to the plans through ongoing interaction and negotiation. Similarly, 
all five of the CCs reported forming their work plans incorporating RCC input 
acquired through either RCC staff surveys or direct communication. In addition, 
all five CCs described working with ED to learn of specific topics and tasks 
needed to advance federal priorities. 

■ Centers conducted work in the topic areas that they identified among the 
objectives in their initial management plans (80 percent of the cases), although 
their activities evolved through the year to address client requests. For example, 
14 of 21 Centers said they conducted work outside their initially planned scope of 
work. Refinements to the work plans included additional work in some topic 
areas, with response to intervention, English language learners, and supplemental 
education services the most common additions. Centers reported that added work 
was in response to emerging needs that were recognized in the process of carrying 
out planned work, and in response to requests from states. Survey data gathered 
from state managers also indicated a strong relationship (.95 correlation) between 
their reported priorities for Center technical assistance and the actual support they 
received. 

■ The kinds of support provided by the RCCs and CCs differed in ways that were 
consistent with the model of technical assistance envisioned by ED. The 



See ehapter 2 for more information about the data sourees and the proeedures for gathering, ending, and analyzing 
the data reported in this ehapter. 
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majority of RCC projects involved front-line assistance activities such as ongoing 
consultation and follow-up (82 percent), whereas this was less so for the work of 
the CCs (22 percent). In contrast, the delivery of research collections and 
syntheses was a central part of the work of the CCs, occurring in 74 percent of 
their projects but 54 percent of the RCC projects. 

■ RCCs were more likely to draw on the work of the CCs than the other way 
around. All 16 RCCs (100 percent) reported drawing on CC expertise, and 48 
percent of the sampled RCC projects had a CC contribution. In comparison, two 
of the five CCs (40 percent) reported that they used RCCs as a source in their 
work. Thirty-seven percent of the sampled CC projects reportedly had some input 
from an RCC; in all cases this included RCC help in identifying participants, and 
in 30 percent of these cases (11 percent of sampled CC projects) it also included 
assistance provision by RCC staff. To some extent, the RCCs and CCs relied on 
their respective networks, with 69 percent of RCCs using the expertise of other 
RCCs and 60 percent of CCs drawing on the knowledge base of other CCs. 

■ External review of Center work was more prevalent among the CCs than the 
RCCs. When asked how they reviewed the content of their technical assistance 
prior to delivery — i.e., what quality assurance process they used — all five CCs 
(100 percent) reported that they submitted all major publications to ED for review 
prior to public release and four of the CCs (80 percent) reported retaining outside 
experts to review draft products. In addition, two CCs described relying on other 
CCs for feedback on their own products as part of the five CCs developing a 
common vetting tool. In contrast, two (12 percent) of the RCCs said they used 
outside experts as a source for vetting. 



Identifying State Needs and Priorities 

A key expectation of the Centers was that they would organize their technical assistance 
work plans around the priorities and needs of the states. In the design of the Center program,"^** 
RCCs had the role of providing “front-line” assistance to a set of state clients and therefore were 
expected to communicate directly with those state agencies. The CCs, on the other hand, were 
charged with translating research and policy for a nationwide clientele and so were expected to 
primarily depend on the RCCs to provide input on aggregated state priorities. The CCs also 
received guidance from ED in order to advance federal priorities. Eor the 2006-07 year, which is 
the subject of this report, the Department approved Centers’ management plans in late spring 
2006. According to the Centers, they gathered information to develop and refine these plans in a 
variety of ways. 

■ RCCs conducted initial needs assessment in consultation with states (exhibit 

3.1); CCs assessed state needs primarily through the RCCs (exhibit 3.2). All 16 



Notice Inviting Applications for New Awards for Fiscal Year 2005. (2005, June 3). Federal Register, 70(106), 
32584. 
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RCCs reported including state input when writing their annual management plans. 
Fifteen of these Centers assessed needs through state client meetings or 
communications, with almost all of them (14 of the 15) communicating directly 
with the chief state school officers. Fewer RCCs conducted surveys of state 
agency staff (10), analyzed state performance data or accountability plans (9), or 
examined state and federal policy environments to identify priorities for technical 
assistance (7). All RCCs were able to conduct needs assessment by building on 
prior contacts with states, either from earlier existing relationships (13) or the 
development of the Center’s grant proposal (1 1). All of the five CCs reported 
forming their work plans incorporating RCC input acquired through either RCC 
staff surveys or direct communication. In addition, all five CCs described ED as 
an influential client with specific requests to focus on certain topics and tasks to 
advance federal priorities such as communicating practical findings from key 
research on ELL or communicating promising practices in high school 
restructuring. Lour of the five CCs also reported defining their initial work 
priorities by examining state or federal policy. 



Exhibit 3.1. RCC strategies for assessing state needs and pianning work 



RCCs 

strategy (N=16) 

Included SEA input when writing annual management plans 16 

Assess needs through meetings or other communication with SEA staff 15 

Communicate directly with the chief state school officer to plan the 
Center’s work 

Drew on SEA relationships or input as part of initial plans for work 16 



Build on relationships with SEAs that existed prior to the 
Center’s grant award 

Included SEA input when writing the original grant proposal or ^ ^ 

cooperative agreement with ED 

Assess needs through survey of SEA staff 1 0 

Assess needs through analysis of state performance data or g 

accountability plans 

Assess needs through analysis of state and federal policy context 7 



EXHIBIT READS: Sixteen RCCs included SEA input when writing annual management plans. 
SOURCE: Center responses to standard response categories during site visits. 
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Exhibit 3.2. CC strategies for assessing RCC and state needs and pianning work 



CCS 

strategy (N=5) 

Assess RCC needs through meetings or other communication with RCC c 
staffs 

Included RCC input when writing annuai management pians 4 

Communicate directiy with the RCC Director to pian the ^ 

Center’s work 

Assess needs through survey of RCC staff 4 

Assess needs through analysis of state and federal policy context 4 

Draw on RCC relationships or input as part of initial plans for work 2 

Buiid on reiationships with RCC staff members that existed prior ^ 

to the Center’s grant award 

Inciuded RCC input when writing the originai grant proposai or ^ 

cooperative agreement with ED 

Assess needs through analysis of state performance data or ^ 

accountability plans 



EXHIBIT READS: Five CCs assessed RCC needs through meetings or other communication. 
SOURCE: Center responses to standard response categories during site visits. 



■ Centers continued to interact with clients to refine their needs assessments and 
negotiate the implementation of their plans (exhibit 3.3). Fifteen of 16 RCCs 
reported forming work groups within state organizations that brought together staff 
from multiple departments to discuss service needs and delivery; the remaining RCC 
reported having worked directly with individual managers such as the chief state 
school officer at the planning stage. Half of the RCCs (8) formed cross-agency work 
groups to discuss common SEA service needs and delivery. All five CCs identified 
needs of their client RCCs primarily by conducting conference calls with designated 
RCC representatives, and three of the five CCs maintained communication about 
needs by forming workgroups that included RCC representatives. In addition, all CCs 
reported providing either large group events or support of existing RCC programs and 
projects as additional ways to learn about and meet both RCC and SEA needs. The 
five CCs maintained ongoing communication with ED regarding federal priorities and 
revisions to their work plans. 
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Exhibit 3.3. Center strategies for ongoing interactions and negotiations 
with ciients 





RCCs 


CCS 


All 


Strategy 


(N=16) 


(N=5) 


(N=21) 


Sponsor large events to make contact with many clients 


13 


4 


17 


Form work groups within client organizations that bring together 
staff from multiple departments/divisions to discuss service 
needs and delivery 


15 


1 


16 


Offer service to support existing client programs/projects/ 
policies 


9 


4 


13 


Form work groups across client organizations to discuss service 
needs and delivery (e.g., different SEAs and/or RCCs) 


8 


3 


11 



EXHIBIT READS: Thirteen RCCs sponsored large events to make contact with many clients. 
SOURCE: Center responses to standard response categories during site visits. 



■ RCCs had direct state interactions to gathering information and keep in touch 
with states’ needs (exhibit 3.4). In interviews, all 16 RCCs identified staff whose 
responsibilities ineluded talking with top- or mid-level SEA managers. For the 12 
multi-state RCCs, staff positions (termed state liaisons) were designated for each 
of the states they served. Of these 12, half hired former SEA employees (one of 
whom was hired in consultation with the state) and 4 embedded the state liaisons 
(two of whom were former SEA employees) within the SEA office space. Among 
the single-state RCCs, one hired a former SEA employee. 



Exhibit 3.4. Formai connections with state clients through RCC staffing 



Type of interaction 


Multi- 

state 

RCCs 

(N=12) 


Single 

state 

RCCs 

(N=4) 


All 

RCCs 

(N=16) 


Has staff whose responsibilities include communication with 
state managers 


12 


4 


16 


Has state-specific staff positions designated for serving as a 
liaison to a state client organization 


12 


1 


13 


Hires former employees of state client organizations 


6 


1 


7 


Has staff positions designated for serving as an in-house 
liaison to a state client organization, maintaining office space 
within the client’s building 


4 


0 


4 


Involves clients in making hiring decisions about Center staff 


2 


0 


2 



EXHIBIT READS: Twelve multi-state RCCs had staff positions designated for serving as a liaison to a 
state client organization. 

SCURCE: Center responses to standard response categories during site visits. 
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Planned and Conducted Center Activities 



To be most helpful to the states, the Centers had to not only plan their teehnieal 
assistanee around the identified priorities but also provide serviees that addressed state needs that 
might shift during the year. Annual management plans written well in advanee might in some 
ways be an asset to the Centers, enabling them to mount serious, eoordinated, sustained efforts 
around their intended programs of work. At the same time, however, the plans might impose 
rigidity on Center teehnieal assistanee, impeding a flexible response to ehanging eireumstanees 
in particular states or nationwide. Whether the Centers delivered the technical assistance states 
asked for and what it looked like can be analyzed from reported Center activities and state 
agency managers’ views. 



Adapting or Adhering to the Management Pian Overaii 

At various points in the program year. Centers were confronted with new client service 
requests (up to 100 new requests as indicated on their quarterly request logs) and learned about 
emerging client needs. As a result. Centers reported ongoing refinements to the general plan of 
work that they had laid out in the management plan. 

■ Most Centers reported providing services and products outside their original 
plan (exhibit 3.5). In retrospective interviews, 14 of the 21 Centers said that they 
conducted services that were outside their initially planned scope of work for the 
year. In fact, nine Centers indicated that they had never declined a client request 
during the year. RCCs indicated that they added work in response to emerging 
needs that they recognized in the process of carrying out planned work, and in 
response to requests from states (commonly new requests for quick-tum-around 
information) that the Centers viewed as consistent with or as an entree toward 
their mission. Added work reported by the CCs was in response to requests from 
ED or the RCCs. Specific requests from ED included work in emerging topic 
areas such as work on response to intervention, high school restructuring, and 
alternate assessments. A total of 6 of the 21 Centers also reported having worked 
with primary service recipients who had not been identified in their planned 
scopes of work; providing assistance to different recipients was more common 
among CCs (four of five, or 80 percent) than among RCCs (2 of 16, or 12 
percent). 

■ More than half (57 percent) of the Centers reported declining some client 
requests (exhibit 3.5). Twelve of the 21 Centers reported having declined to 
provide particular technical assistance services. Among the 12 Centers that 
declined work, 8 indicated that they, at some point, had discussed with their 
clients (i.e., RCCs with states and CCs with RCCs) what they were and were not 
authorized to do, based on the legislation and the original grant Notice Inviting 
Applications (e.g.. Center declined request to work with districts directly). 

Eight Centers reported having declined requests because they judged that the 
work wasn’t on target with the best use of their resources, particularly as it related 
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to building capacity (e.g., Center deelined request to provide quiek-fix help rather 
than longer-term eapaeity building); eight indieated they had deelined requests 
beeause the work fell outside the parameters of their eooperative agreement with 
ED (e.g., Center declined request to work on topics that were narrow and/or 
unrelated to NCLB); and four Centers cited both of these reasons for having 
deelined requests. 



Exhibit 3.5. Center strategies for adapting or adhering to the pian of work 





RCCs 

(N=16) 


CCS 

(N=5) 


All 

(N=21) 


Refer to parameters set by NCLB or the Center’s authorizing 
legislation when negotiating with clients to plan its work 


13 


2 


15 


Provide service outside of initial planned scope of work 


10 


4 


14 


Work on topics outside of pian ned scope of work 


10 


3 


13 


Work with primary service recipients outside pianned 
scope of work 


2 


4 


6 


Decline to provide a service 


9 


3 


12 


Center has a staff member and process for assuring that 
Center work is within its management plan 


9 


2 


11 



EXHIBIT READS: Thirteen RCCs referred to parameters set by NCLB or the Centers’ authorizing legislation 
when negotiating with clients to plan their work. 

SOURCE: Center responses to standard response categories during site visits. 



Technicai Assistance Delivered, in Comparison with Initial Plans and with 
Reported State Priorities 

One way to measure the stability and flexibility in the Centers’ work is to assess the 
extent to which Centers followed or adjusted their initial plans with respeet to the topieal 
eoverage of the teehnical assistance. We examined Center perspectives to indieate the extent to 
whieh initially planned teehnical assistance matched the technical assistance the Centers 
delivered, by topie. The states’ reports on their priorities and their reeeipt of Center technieal 
assistance indicated the extent to which the teehnieal assistanee actually delivered was 
responsive to state priorities. 

■ State survey data revealed a strong association between their priorities and their 
receipt of technical assistance from the Centers, although for each priority area 
there were states that wanted technical assistance but did not receive it. Through 
a survey, managers in all the SEAs were asked to identify their technical 
assistance priorities from a list of state responsibilities in the No Child Left 
Behind (NCLB) legislation. The bar chart in exhibit 3.6 eompares the percentage 
of all state managers who expressed a priority for teehnical assistance on a 
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particular NCLB responsibility with the percentage reporting that Centers 
provided them with major or moderate teehnical assistanee on those 
responsibilities. For example, 98 percent of state managers reported that building 
or managing statewide systems of support was a major or moderate priority for 
Center teehnical assistance, and 72 percent said that they reeeived a major or 
moderate amount of Center teehnieal assistanee in this area."^' An analysis of the 
similarity of the distributions of the two rankings of the 1 1 areas shows a 
statistically significant relationship, with a correlation coeffieient of +0.95 (using 

42 

Spearman’s rho). 



Exhibit 3.6. State priorities for technicai assistance and receipt of Center 
technicai assistance, by area of NCLB responsibiiity 

Building or managing a statewide system of 
support 

Training or managing the state-level staff or 
school support teams 

Identifying and/or developing programs or 
models that address district and/or school needs 

Designing or implementing state assessment and 
accountability systems 

Formulating or refining state policies to respond 
to NCLB requirements 

Supporting use of assessment data by schools and 
districts 

Disseminating information on scientifically-based 
research to districts and schools 

Providing training and professional development 
to local educators in academic subjects 

Aligning state accountability systems with NCLB 
accountability systems 

Monitoring compliance with NCLB requirements 
in districts and schools 

Communicating with the public about NCLB 
requirements or report cards 

0 20 40 60 80 100 

Percent 

EXHIBIT READS: Ninety-eight percent of states reported that SSOS was a major or moderate priority for technical 
assistance. Seventy-two percent of states reported that they received a major or moderate amount of technical 
assistance in that area. 




Among the subgroup of state managers who reported a major or moderate priority for any particular state 
responsibility, at least 69 percent reported receiving some technical assistance with that responsibility from the 
Centers (a finding discussed in chapter 4; see exhibit 4.5). 

The Spearman’s rho correlation statistic measures the degree of monotonic relationship between two variables 
that are arranged in rank order on ordinal scales. It differs from Pearson's correlation only in that the computations 
are done after the numbers are converted to ranks. It is a nonparametric statistic, and does not require that the 
distribution of the individual measures meet assumptions of normalcy or be interval data. 
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SOURCE: Survey of State Managers. Responses weighted so that each state was represented by a composite in 
cases where more than one manager from that state responded 

■ Centers reported providing technical assistance in a majority (80 percent) of the 
instances where identified topic areas were included in their management plans 
(exhibit 3. 7). Comparing the contents of the management plans with the annual 
PIFs) where Centers reported their delivered work to the evaluation team, of the 
133 topic-related objectives in the plans, the Centers provided services and 
products on 107 of them. The fact that some planned cases"^^ (133-107=26) were 
not listed on the PIFs indicates that 19 percent of the cases (26 of the planned 
133) were not conducted. Looking at the work actually conducted, in 38 percent 
of cases ([172-107]/! 72) the Center provided technical assistance on a topic not 
initially cited in its planned objectives. 

■ The work of the CCs was spread across the topic areas (exhibit 3.8). Only in the 
area of English language learners did more than one CC (2) plan and deliver work 
products or services. This is consistent with the intent of ED in defining specific 
areas of expertise for each of the five CCs. 

■ Centers commonly expanded the topic areas their technical assistance covered 
by adding work on topics they did not initially include in their plans. Eooking at 
exhibit 3.7 and comparing columns 3 and 2, there are 19 of the 22 topics in which 
technical assistance was not included in a Center’s management plan but was 
delivered by that Center. The most common shift toward topics were in the areas 
of response to intervention (seven Centers conducted work in this area that was 
not planned), English language learner issues (five Centers), highly qualified 
teacher provisions of NCEB (five Centers) and Supplemental Educational 
Services (five Centers). Based on exhibit 3.8, addition of work was more common 
among RCCs than CCs (in 19 of 22 topics, RCCs that did not have an objective in 
a topic conducted work on that topic; in comparison, in 7 of 22 topics CCs 
conducted work on a topic that that Center had not anticipated in its annual 
management plan). 



Using each Center’s management plan, the evaluation team coded for each Center whether it had specifically 
stated an intention/objective to address work in each of the 22 topics (i.e., the first column of numbers in exhibit 3.7 
indicates the number of Centers that stated an objective of conducting any work in each of the 22 topics, beginning 
with the topic of state systems of support). Using the PIFs, the evaluation team coded project topics to determine the 
number of Centers that actually did work in each of the 22 topics (i.e., the last column of numbers, by topic, in 
exhibit 3.7). 
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Exhibit 3.7. Topics on which Centers stated objectives and/or deiivered projects 







Number of Centers 

(N=21) 




Topic 


Objective on 
topic was 
stated in 
management 
pian 
(1) 


Objective on topic was 
stated in management pian 
AND at ieast one 
project on topic was 
reported on PiF 
(2) 


At ieast one 
project on 
topic was 
reported on 
PiF 
(3) 


Total cases of a Center setting an 
objective and/or reporting a project 


133 


107 


172 


Components of effective systems of 
support — state, district, school 


15 


15 


17 


English language learners 


10 


10 


15 


Flighly qualified teacher provisions of NCLB 


8 


5 


10 


Teacher professional development 


6 


5 


8 


Teacher preparation and induction 


5 


3 


5 


Data use/data-driven decisionmaking 


8 


4 


7 


Assessment design 


6 


5 


5 


Formative assessment 


5 


4 


7 


Special education — curriculum, instruction, 
and professional development 


8 


3 


6 


Response to Intervention (Rtl) 


3 


3 


10 


Special education — assessment 


3 


3 


3 


High school redesign/reform 


8 


7 


10 


Transition to high school 


2 


2 


5 


Dropout prevention 


2 


2 


2 


Mathematics 


7 


4 


8 


Adolescent literacy 


6 


6 


8 


Reading 


6 


5 


8 


Supplemental educational services (SES) 


7 


7 


12 


Parent involvement 


7 


7 


11 


Migrant education 


5 


3 


6 


Data management compliance 


5 


3 


5 


Indian/Native American education 


1 


1 


4 



NOTE: Column (1) - column (2) =the number of Centers that deleted work in the topic area as indicated by 
number of Centers that had a topic-related objective in their management plan minus the number of Centers that 
had the topic area on their PIF and their management plan; column (3)- column (2) =the number of Centers that 
added work in the topic area as indicated by number of Centers that reported work on their PIF minus the 
number of Centers that had the topic area on their PIF and their management plan. 

EXFIIBIT READS: For the topic “Components of effective systems of support — state, district, school,” 15 Centers 
had a related objective in their respective annual management plans; of these, all 15 reported projects on the 
topic in their PIF; and a total of 17 Centers had projects on the topic, whether or not they had stated an objective 
related to it in their management plan. 

SCURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation 
team. 
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Exhibit 3.8. Topics on which RCCs and CCs stated objectives and/or delivered 
projects 







Number of RCCs (N=16) 






Number of CCs (N=5) 






Objective 
on topic in 
manage- 
ment plan 
(1) 


Objective on 
topic in plan AND 
project(s) on 
topic on PIF 
(2) 


Project(s) 
on topic on 
PIF 
(3) 


Objective 
on topic in 
manage- 
ment pian 
(4) 


Objective on 
topic in plan AND 
project(s) on 
topic on PIF 
(5) 


Project(s) 
on topic on 
PIF 
(6) 


Total cases 


112 


89 


146 


21 


18 


26 


Components of effective 
systems of support — state, 
district, school 


14 


14 


15 


1 


1 


2 


English language learners 


8 


8 


12 


2 


2 


3 


Fllghly qualified teacher 
provisions of NCLB 


7 


4 


9 


1 


1 


1 


Teacher professional 
development 


4 


4 


7 


2 


1 


1 


Teacher preparation and 
Induction 


4 


2 


4 


1 


1 


1 


Data use/data-drIven 
decisionmaking 


7 


3 


5 


1 


1 


2 


Assessment design 


5 


4 


4 


1 


1 


1 


Formative assessment 


4 


3 


5 


1 


1 


2 


Special education - 
curriculum. Instruction, 
professional development 


6 


3 


5 


2 


0 


1 


Response to Intervention 
(Rtl) 


3 


3 


8 


0 


0 


2 


Special education- 
assessment 


2 


2 


2 


1 


1 


1 


Fligh school 
redesign/reform 


7 


6 


9 


1 


1 


1 


Transition to high school 


1 


1 


4 


1 


1 


1 


Dropout prevention 


1 


1 


1 


1 


1 


1 


Mathematics 


6 


3 


7 


1 


1 


1 


Adolescent literacy 


5 


5 


6 


1 


1 


2 


Reading 


5 


4 


7 


1 


1 


1 


Supplemental educational 
services (SES) 


6 


6 


11 


1 


1 


1 


Parent Involvement 


6 


6 


10 


1 


1 


1 


Migrant education 


5 


3 


6 


0 


0 


0 


Data management 
compliance 


5 


3 


5 


0 


0 


0 


Indian/Native American 
education 


1 


1 


4 


0 


0 


0 



NOTE: column (1 )-column (2) =the number of RCCs that deleted work In the topic area as Indicated by number of Centers that 
reported work on their management plan minus the number of Centers that had the topic area on their PIF and their management 
plan ; column (3)- column (2) =the number of RCCs that added work In the topic area as Indicated by number of Centers that 
reported work on their PIF minus the number of Centers that had the topic area on their PIF and their management plan. These 
calculations are analogous for the CCs. 

EXFIIBIT READS: For the topic “Components of Effective Systems of Support — State, District, School,” 14 RCCs reported a related 
objective In their management plans; of these, all 14 reported projects on the topic; and a total of 15 RCCs had projects on the topic, 
whether or not they had originally stated an objective related to It. 

SCURCES: Center management plans for 2006-07; PIFs prepared by Centers In consultation with evaluation team. 
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■ Centers also shifted away from providing work on originally planned topics. A 
comparison of columns 2 versus 1 in exhibit 3.7 indicates that in 10 of the 22 
topics, Centers that set an objective in a topic did not deliver a project related to 
the topie — in other words, there was a shift away from initially stated objectives 
on these topies. The two topies in whieh the largest number of Centers planned to 
do work but did not deliver a projeet were speeial education curriculum, 
instruction, and professional development (five Centers); and data use or data- 
driven deeision making (four Centers). As seen in exhibit 3.8, deletion of work 
was more common among RCCs than CCs (in 11 of 22 topics, one or more RCCs 
that reported having an objective in a topic in its management plan did not report 
a project on the topie in its PIF; in eomparison, in 2 of the 22 topic areas, CCs that 
had planned work did not deliver a project in that topic). 

It is not possible to ascertain from the data whether these shifts from initial objeetives to 
projects delivered took plaee at the request of elients. Based on interviews with the Centers, 8 of 
the 16 RCCs said they changed the work they planned to deliver during the year beeause of 
ehanges in SEA priorities or leadership. One CC similarly said that, as a result of turnover of a 
key client in the state, it had to put on hold collaborative work it had with an RCC that was to be 
delivered to the SEA. In addition, the ratings of relevance and usefulness that state managers 
give to the Centers’ work (see chapter 4) provide an indication that the changes in work plans 
may have been in response to evolving state needs. 



Activities and Products Provided by the Centers 

Regardless of whether they were originally speeified in work plans or added later, what 
types of technieal assistance did the Centers provide? As described above, the most common 
topics on which Centers reported delivering assistance were systems of support, English 
language learners, high school redesign/reform, supplemental education services, and parent 
involvement (exhibit 3.7). 

An analysis of 122 sampled projects (described in more detail in ehapter 2) provides 
more in-depth information about the nature of Center technieal assistanee aetivities or resourees. 
Although the sample of projects is not statistically representative of the Centers’ work, recall that 
project sample selection for review of each Center’s work favored their most dominant projects 
and included over half (56 pereent) of that year’s designated major or moderate projeets. A 
project is defined for this evaluation as “a group of closely related aetivities and/or deliverables 
designed to aehieve a speeific outcome for a speeific audience.” 

■ Most (1 02 of 122 = 84 percent) projects included the delivery of more than one 

type of activity or resource (exhibit 3.9). Projects with a single eoded type of 
activity/resouree were more common among the CC than the RCC sampled 
projects (9 of 27 = 33 percent and 11 of 96 = 11 percent, respectively), illustrating 
the differing nature of their work. The evaluation team coded the following seven 
types of activities and resourees that Center projects could provide; ongoing 
eonsultation and follow-up, researeh eollections and syntheses, engagement of 
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participants in project planning, training events, task foree meetings and work, 
eonferenees, and support for development of a formal plan to implement a 
program or poliey. 



Exhibit 3.9. Sampied Center projects by number of activity or resource types 
provided 



Number of activity or 
resource types: 


RCC 

projects 

(n=96) 


CC 

projects 

(n=27) 


Aii 

projects 

(n=122) 


Totai projects 


96 


27 


122 


One 


11 


9 


20 


Two 


22 


8 


30 


Three 


26 


3 


29 


Four 


16 


4 


19 


Five 


16 


3 


19 


Six 


5 


0 


5 



EXHIBIT READS: Eleven RCC projects provided a single type of activity or resource for participants. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. The 
total number of projects was 122. One project collaboratively conducted by an RCC and a CC was 
counted among both RCC projects and CC projects but was only counted once among the projects of 
all Centers. 

■ The kinds of support provided by the RCCs and CCs differed in ways that are 

consistent with the model of technical assistance envisioned by ED 
(exhibit 3.10). The guidance given by ED through the Center grant competition 
and afterwards laid out a particular structure for the Centers’ work: CCs would 
specialize in activities that require content specialization while the RCCs would 
specialize in state client interactions. The majority of RCC projects involved 
front-line assistance activities such as ongoing consultation and follow-up (79 of 
96 projects, or 82 percent), whereas this was less so for the work of the CCs (6 of 
27 projects, or 22 percent). In contrast, research collections and syntheses were a 
central part of the work of the CCs, occurring in 74 percent of their projects but 
54 percent of the RCC projects. The second most common project activity for 
CCs (63 percent) was conferences, an activity that also was consistent with the 
content specialization model promoted by ED. 
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Exhibit 3.10. Sampied Center projects by types of participant activities and 
products 



Activities and products (with clarifying definitions used by 
coders) 


RCC 

projects 

(n=96) 


CC 

projects 

(n=27) 


Aii 

projects 

(n=122) 


Ongoing consultation and follow-up (multiple contacts to same 
participants, that were part of a coherent and purposeful whole) 


79 


6 


84 


Research collections and syntheses 


52 


20 


71 


Engagement of participants in project planning (more than needs 
assessment or identifying participants) 


43 


8 


50 


Training events (focused on implementing a specific program or 
strategy) 


41 


10 


50 


Task force meetings and work (focused on addressing a specific 
problem, program, or policy) 


48 


2 


50 


Conferences (symposium, forum, institute; highlights a range of 
perspectives, strategies, or programs) 


26 


17 


43 


Support development of a formal plan to implement a program or 
policy 


18 


2 


20 



EXHIBIT READS: Seventy-nine RCC projects included ongoing consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. The total number 
of projects was 122. One project collaboratively conducted by an RCC and a CC was counted among both RCC 
projects and CC projects but was only counted once among the projects of all Centers. 



Knowledge Base Used by Centers 

An important part of Centers’ operations is how and to what extent they draw on 
expertise to provide a solid foundation for recommended strategies and practices. We asked 
Centers about the sources they drew on in developing or reviewing their technical assistance, 
including their reliance on other Centers as sources. 



Sources of Content Expertise 

The analysis investigated the mix of in-house sources, scholarly sources, federally supported 
resources of various kinds, and state-level practitioners that were sources for each type of Center. A 
particular focus was the extent to which the RCCs used CC expertise, since that was a key 
component of how ED thought the Center system would operate. Of additional interest was the 
extent to which the CCs enlisted the RCCs or states as a source of knowledge since the Centers 
were charged with identifying and disseminating promising practices (exhibit 3.11). 

■ All 16 RCCs reported that CC expertise contributed to the knowledge base for 
their products and services, a finding consistent with the program design. To 
communicate with CCs and stay abreast of their activities, based on interviews 
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with the Centers, all 16 RCCs reported using the CCs in developing their produets 
and serviees. In faet, the RCCs reported designating staff to serve as liaisons to 
the CCs, and all five CCs deseribed aetivities that involved these liaisons. This 
organizational strategy may have contributed to RCC use of CC expertise. 

■ The CCs were less likely to draw on the RCCs as a source of expertise 
(exhibit 3.10). Two of the five CCs (40 percent) reported that they relied on RCCs 
in the process of developing their materials or services. 

■ Every RCC and every CC reported using in-house expertise from its own staff. 
For all Comprehensive Centers, the grantees and/or subgrantees were affiliated 
with larger organizations that also included personnel who were not officially part 
of the Center (all grantees and subgrantees are listed in appendix A). Of the 16 
RCCs, 15 used the expertise of non-Center staff in their lead or subgrantee 
organizations, 13 used staff in their lead organization and 10 in their subgrantee 
organizations. Of the five CCs, three reported using non-Center staff from lead 
and subgrantee organizations. 

■ After drawing on their own organizations, the RCCs and CCs relied on different 
sources. Other than using the CCs, the RCCs most often reported turning to 
professional associations and other RCCs as sources of expertise (each type of 
source was reported by 1 1 of the 16 RCCs). The source most frequently cited by 
CCs, after internal Center staff, was consulting firms or other private contractors 
(four of the five CCs). 

■ RCCs were more likely than CCs to draw on state agency expertise. Because 
Centers were charged with identifying and disseminating promising practices 
from practitioner sources, especially states, the evaluation assessed the extent to 
which knowledge flowed to the Centers from their state clients. Ten of the 16 
RCCs (62 percent), those charged with working directly with the states, identified 
SEAs as a source of expertise. In contrast, one of five CCs (20 percent) identified 
SEAs as a source of expertise. Given the Center structure of the CCs working 
through the RCCs, the CCs could learn about state practices through the RCCs; 
however, three of the five CCs did not indicate RCCs as a source. 

■ Regional Educational Laboratories, with which the Centers were expected to 
cooperate, were reported as a source of expertise by 10 of the 16 RCCs and 1 of 
the 5 CCs. 
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Exhibit 3.11. Sources of content expertise used in deveioping Center products 
and services 



Source 


RCCs 

(N=16) 


CCs 

(N=5) 


All 

(N=21) 


Internal Center staff (lead organization and/or subgrantees) 


16 


5 


21 


Content Center(s) (CCs) 


16 


3 


19 


Other staff from lead and subgrantee organizations, who are not 
formally employed by the Center 


15 


3 


18 


Other staff in the lead organization 


13 


2 


15 


Other staff in subgrantee organization(s) 


10 


2 


12 


Professional associations (e.g., AERA, CCSSO) 


11 


3 


14 


Consulting firms or private contractors 


9 


4 


13 


Regional Center(s) (RCCs) 


11 


2 


13 


Other federally funded technical assistance providers (not part of the 
Center) 


8 


3 


11 


Regional labs (not part of the Center) 


10 


1 


11 


State education agencies (SEAs) 


10 


1 


11 


Universities 


7 


3 


10 


What Works Clearinghouse 


7 


1 


8 



EXHIBIT READS: All sixteen RCCs relied on internal Center staff as sources of content expertise used in 
Center products and services. 

SOURCE: Center responses to standard response categories during site visits. 



Extent of RCC-CC Coordination 

As part of the assessment of the way the Centers funetioned as a system, the evaluation 
also looked at the extent to which the two types of Centers coordinated with or informed each 
other, both at the Center level (as described above, in the description of Centers’ reliance on one 
another as sources of expertise) and also at the project level. Thus, in the interviews Centers were 
asked how they informed each other of their work. The evaluation team also identified the 
sampled RCC projects in which a CC was a source of materials, advice, or in-person assistance 
to clients, and the CC projects in which an RCC participated by recruiting participants, brokering 
service, or delivering assistance to clients. 

■ Periodic conference calls and specific RCC requests to CCs were the primary 
mechanism for coordination between CCs and RCCs. All five CCs offered 
regular conference calls for their designated liaisons at the RCCs in order to 
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convey their work to the RCCs. In addition, all CCs used these conference calls, 
sometimes in combination with webinars, to conduet RCC staff training and to 
identify staff needs. All RCCs reported making direct requests for CC assistanee 
when they felt it would be useful. Finally, ED sponsored bi-annual Directors’ 
meetings that provided opportunities for RCCs and CCs to eoordinate. 

■ Some informal coordination took place among RCCs. Although ED has few 
formal mechanisms for RCCs to share information about their work with each 
other, all RCCs reported in interviews that they eommunieated with other RCCs, 
most often those with which they had formal organizational relationships (i.e., the 
same parent organization, or a subgrant with the other RCC’s parent 
organization). Reports of viewing other RCC websites, registering for webinars, 
and direct phone conversations were additional methods staff mentioned as 
informal ways to learn about other RCC work. 

■ Half (48 percent) of the RCC projects had a CC contribution of one or more 
kinds (exhibit 3.12). Eorty-six of the 96 projects were found to have had input 
from a CC. The most frequent type of CC contribution to RCC projects was in 
material support (42 projects), including practice briefs, researeh syntheses, and 
implementation manuals. Seventeen RCC projects included CC staff providing 
technical assistance directly to project participants, such as to state workgroups or 
at conferenee events. 



Exhibit 3.12. Sampied RCC projects by CC contribution 



RCC 

projects 



CC contribution (n=96) 

RCC project had no CC contribution 50 

RCC project had a CC contribution 46 

CC provided materials used in this RCC project 42 

CC provided advice to the RCC on this project 1 6 

CC delivered technical assistance to project participants 1 7 

EXHIBIT READS: In 50 RCC projects, a CC did NOT contribute. 



SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. 



■ A smaller proportion of CC projects had an RCC contribution of one or more 
kinds (exhibit 3.13). Ten of 27 sampled CC projects were documented to have 
input from the RCCs. In each of these 10 projects, RCCs recruited participants or 
brokered CC services. In 3 of them, or 1 1 percent of the sample of 27 CC 
projects, RCCs also delivered assistance to project participants (e.g., presented at 
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a conference, contributed content for a product, delivered follow-up assistance 
that continued work initiated by a CC). 

Exhibit 3.13. Sampied CC projects by RCC contribution 



CC 

projects 



RCC contribution (n=27) 

CC project had no RCC contribution 17 

CC project had an RCC contribution 10 

RCC recruited participants or brokered service 1 0 

RCC deiivered technicai assistance to project participants 3 

EXHIBIT READS: In 17 CC projects, an RCC did NOT contribute. 



SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. 



Product Quality Control Procedures 

The evaluation inquired about the proeedures and types of individuals brought to bear in 
the proeess of reviewing produets before release. Given that project quality is assessed as part of 
the evaluation, it is useful to understand the processes that the Centers themselves reported using 
to promote quality of the products used in their technical assistance. 

■ Most RCCs and all CCs relied, in part, on internal staff for quality control 
(exhibit 3.14). When asked about how they reviewed the content of their technical 
assistance prior to delivery — i.e., what quality assurance process they used — 14 of 
the 16 RCCs and all five of the CCs described processes that included internal 
staff review of their work. Of the two that did not describe a vetting process that 
included internal staff, one relied on an external source and another did not clearly 
identify a process. 

■ CCs were more likely than RCCs to report having external sources review their 
products (exhibit 3.14). All five CCs (100 percent) reported that they submitted 
all major publications to ED for review prior to public release, as they were 
required to do. Also, all five CCs (100 percent) reported using other external 
sources for review, including four CCs (80 percent) that reported retaining outside 
experts to review draft products and two CCs (40 percent) that described relying 
on other CCs for feedback on their own products as part of the five CCs 
developing a common vetting tool. In contrast, two (12 percent) of the RCCs said 
they used outside experts as a source for vetting. As important context, the nature 
of the products and services conducted by the RCCs and the CCs differs, with the 
RCC work consisting of fewer formal products than that of the CCs. 
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Exhibit 3.14. Sources for vetting Center products and services prior to deiivery 





RCCs 


CCS 


All 


Source 


(N=16) 


(N=5) 


(N=21) 


Internal Center staff (lead organization and/or subgrantees) 


14 


5 


19 


Other staff from lead and subgrantee organizations, not formally 


6 


Q 


9 


employed by the Center 


O 


Other staff in the lead organization 


5 


3 


8 


Other staff in subgrantee organization(s) 


5 


2 


7 


External sources 


10 


5 


15 


U. S. Department of Education 


6 


5 


11 


Content Center(s) 


6 


2 


8 


Outside experts retained to review drafts 


2 


4 


6 



EXHIBIT READS: Fourteen RCCs relied on internal Center staff as sources for vetting Center products prior to 
release. 

SOURCE: Center responses to standard response categories during site visits. 
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4. State Managers’ Assessment of the Centers’ 
Technical Assistance 



In any evaluation, performance is judged by whether the program meets its goals and 
objectives. The Comprehensive Centers were expected to provide technical assistance that would 
build states’ capacity to carry out their responsibilities under NCLB, supporting district and 
school efforts to close achievement gaps and raise student achievement. Given the diffuse and 
indirect relationship between state-level technical assistance and student outcomes, it is 
impossible to attribute changes in test scores to the work of the Centers. It is possible, however, 
to determine whether state agency managers felt the technical assistance addressed their 
agencies’ purposes and expanded agency capacity. 

This chapter examines how the work of the Comprehensive Centers was regarded by key 
clients, specifically senior managers in state education agencies. Although the assistance might 
vary in quality or utility across Centers or projects (see chapter 5), as a first step it is important to 
understand how states viewed the output of the Comprehensive Centers program in a broad 
sense. We asked state managers whether the technical assistance provided by their RCC and the 
Content Centers served their agency’s purposes, whether it had addressed each of several 
specific areas of state responsibility under NCLB, to what extent it had expanded state capacity, 
and how it compared with technical assistance from other sources. 

The key findings include: 

■ Eighty-eight percent of state managers (weighted/^* reported that the Centers’ 
technical assistance was at least “a good start” toward serving state purposes, 
with 36 percent overall indicating that the Centers’ assistance “served the 
state’s purposes completely. ” Managers who said the state’s purposes were not 
completely served reported a variety of different issues. For 33 percent of them 
(or 21 percent of all state managers), the Centers were too slow in getting 
assistance up and going; 33 percent (21 percent overall) reported that the state’s 
most important priorities for technical assistance were outside the scope of the 
Centers’ work; and 31 percent (20 percent overall) reported that the state secured 
most of its technical assistance from sources other than the Centers. 

■ For each of the four areas of NCLB implementation most widely identified as 
state priorities for technical assistance (see chapter 3), at least 90 percent of the 
state managers who had identified that area as a priority had received 
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For 20 states, the evaluation team reeeived eompleted responses from more than one state manager. As the state 
was the primary unit of analysis in analyzing data from the state manager survey for this report, the state managers’ 
responses were weighted to ensure that eaeh state was equally represented in all summary statisties while taking into 
aeeount the variation in responses within eaeh state. The weighting proeedure, where eaeh response was weighted 
by the inverse of the number of managers responding from that state, ensured that eaeh state was equally represented 
when the team aggregated responses aeross states to deseribe the distribution of responses. See ehapter 2 for a 
detailed deseription of the sampling and analysis proeedures. 
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assistance with it from the Centers. In the area of building or managing a 
statewide system of support, 98 pereent of the state managers who identified it as 
a priority said the Centers provided teehnieal assistanee with it; the eorresponding 
figures were 95 pereent for identifying programs or models for distriets or 
sehools, 90 pereent for training or managing state support teams, and 97 percent 
for disseminating information on scientifically based research. 

■ Most (68 percent) of the state managers reported that Center assistance had 
“greatly” expanded their agency’s capacity in at least one of 11 areas of NCLB 
responsibility about which they were surveyed. 

■ State managers reported that the Comprehensive Center network was one of 
multiple sources that they used for technical assistance. On average, state 
managers ranked the Centers as one of the top three sources of technical 
assistance that they relied upon, along with professional associations and the ED- 
funded Regional Educational Eaboratories. The purposes for which states used the 
Centers more than other sources were “to plan the initial steps in solving a 
problem” (reported as a purpose for Center technical assistance by 66 percent of 
state managers) and “to develop the skills of SEA or intermediate education 
agency staff’ (61 percent of state managers). 

■ Overall, on a scale of l(very low degree) to 5 (very high degree), state managers 
gave the Centers’ early technical assistance close to a 4. The average score for 
‘relevance” was 3.94 and the average score for the “usefulness” of the Center’s 
work was 3.86. These scores fell just below the 4.0 threshold defined as “to a high 
degree” on the 5 -point scale that state managers used to determine their responses. 



Extent to Which Center Assistance Served State Purposes 

ED placed a significant emphasis on having the Centers, both RCCs and CCs, deliver 
technical assistance that would advance state efforts to implement NCEB. In developing their 
work plans and delivering technical assistance, the Centers were expected to target their work on 
state concerns and priorities, as discussed in chapter 3. States’ views of how well the assistance 
met their own purposes provided one perspective on the success of the program. 

■ More than one third (36 percent) of state managers reported that Center 

technical assistance “served the state’s purposes completely,” with another 52 
percent reporting that “it was a good start. ” Six percent said the technical 
assistance “was a start, but some important priorities were not addressed”; the 
same percentage said that, “for the most part, it did not serve the state’s purposes” 
(exhibit 4.1). 
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Exhibit 4.1. Extent to which technicai assistance from the Centers served state 
purposes, as judged by state managers 



Extent 


Percent of state 
managers (weighted) 

(n=56) 


It served the state’s purposes completely 


36 


It was a good start 


52 


It was a start, but some important priorities were 
not addressed 


6 


For the most part, it did not serve the state’s 
purposes 


6 


EXHIBIT READS: Thirty-six percent of state managers (weighted) reported that Center 
technical assistance served the state’s purposes completely. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally 
represented in instances where more than one manager from a state responded. 



■ The two most common reasons cited for the Centers not fully serving state 
purposes were the length of time it took to get projects underway and the 
identification of state priorities that were outside the Center’s line of work. The 
survey posed a series of follow-up questions asking why, to the respondents who 
did not report that Center assistanee “served the state’s purposes eompletely” 
(exhibit 4.2). These state managers were asked to eheek off as many as eight 
possible reasons that the assistanee was less helpful to their state than it might be. 
Among these 30 state managers, more than half (56 pereent) identified a single 
reason, a quarter (27 pereent) identified two reasons, and 15 pereent three or more 
reasons. Thirty-three pereent (21 pereent of all state respondents) eited (1) the 
proeess of negotiating a work seope and organizing projeets took too long, and (2) 
the state’s most important priorities for teehnieal assistanee fell outside the 
Center’s seope of work. Out-of-seope requests for teehnieal assistanee would 
inelude, for example, working direetly with loeal distriets and sehools. 

■ The next most common reasons reflected states’ use of or desire to use, sources 
other than the Comprehensive Centers for technical assistance. In 3 1 pereent of 
states whose purposes were not fully served (20 pereent overall), managers 
reported that the state filled most of its teehnieal assistanee requirements from 
other sourees (exhibit 4.2). Asked whether the state would prefer to eontraet 
direetly with experts and eonsultants rather than reeeiving teehnieal assistanee 
from the Comprehensive Centers, 19 pereent of these respondents (12 pereent of 
state managers overall) agreed. The survey responses of these managers did not 
suggest a high level of dissatisfaetion with the Center assistanee or a heavy 
relianee on other sourees, however. Among those who had given either or both of 
these explanations for less than eomplete satisfaetion with Center assistanee, 50 
pereent nevertheless reported that they used the Centers “to a great extent.” There 
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was no other source of assistance that this group of managers reported using more 
than they used the Centers, and no source that they reported using more than the 
other state managers did. 



Exhibit 4.2. Reasons why Center assistance served some states’ purposes 
iess than compieteiy 





Percent of those state managers 
(weighted) who reported Center 
assistance had not served their 
purposes compieteiy (n=36) 


Reasons 


Yes 


No 


The process of negotiating a work scope and 
organizing projects takes too long 


33 


67 


The state’s most important priorities for assistance 
fall outside the Comprehensive Center’s scope of 
work 


33 


67 


The state secures most of the technical assistance it 
needs from other sources 


31 


69 


The state would prefer to locate and contract directly 
with the experts or consultants from whom it needs 
assistance, rather than working through the 
Comprehensive Centers 


19 


81 


Comprehensive Center staff are not able to spend as 
much time working with the state as we would like 


17 


83 


The Comprehensive Center does not have the 
expertise the state needs 


15 


85 


A policy or priority shift at the state level caused the 
Center’s assistance to be less helpful than it might 


7 


93 


The state has been unable to develop a productive 
working relationship with the Center 


3 


97 



EXHIBIT READS: Of the 36 state managers (weighted) reporting Center assistance served their state’s 
purposes less than completely, 33 percent said a reason was that the process of negotiating a work scope 
and organizing projects takes too long. 

SOURCE: Survey of State Managers. Respondents limited to those who did not answer “It served our 
purposes completely’’ to the preceding question. Responses weighted so that each state was equally 
represented in instances where more than one manager from a state responded. 



Reported Assistance Delivery and Capacity Building for State 
NCLB Responsibilities 

The primary aim of the Center program was to deliver technical assistance that would 
both address state priorities in implementing NCLB and build state capacity for NCLB 
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implementation. Senior state managers reported their pereeptions of the extent to whieh the 
Centers had delivered assistanee and built eapaeity in the areas of NCLB responsibility that were 
teehnieal assistanee priorities for their state. 



■ For each of the four areas of NCLB implementation most widely identified as 
state priorities for technical assistance, at least 90 percent of the state managers 
who identified that area as a priority had received assistance with it from the 
Centers. The eorresponding pereentage reporting at least a moderate level of 
assistanee was at least 74 pereent (exhibit 4.3) for eaeh of four areas (building 
statewide systems of support, identifying programs, training statewide support 
teams, and disseminating information on soientifieally based researeh). In the area 
of building or managing a statewide system of support, 98 pereent of the state 
managers who identified it as a priority said the Centers provided teehnieal 
assistanee with it. 

■ There were four areas in which the Centers reportedly had not delivered 
assistance to 20 percent or more of the state managers for whom the area was a 
priority. These areas were eommunieating with the publie (an area in whieh 3 1 
pereent of the 16 states identifying it as a priority had not reeeived Center 
assistanee), monitoring eomplianee with requirements (29 pereent of 24 states), 
providing training to loeal edueators in aeademie subjeets (21 pereent of 29 
states), and designing or implementing state assessment or aeeountability systems 
(20 pereent of 25 states) (exhibit 4.3). These four areas were also among the least 
widespread priorities for teehnieal assistanee among the states: the only one that 
was a moderate or greater priority for more than half of the states was training for 
loeal edueators in aeademie subjeets. 
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Exhibit 4.3. State priorities for technicai assistance and assistance 
received from Comprehensive Centers 





Among state managers reporting 
the responsibility as a major or 
moderate priority, the percent 
reporting they received: 


Priority area of state responsibility under NCLB 


Any assistance 
from the Centers 
related to the 
responsibility 


Major or 
moderate 
assistance from 
the Centers 
reiated to the 
responsibiiity 


Building or managing a statewide system of support for 
districts and schools identified for improvement under NCLB 
(n^46) 


98 


80 


Training or managing the state-level staffer school support 
teams who provide support to districts and schools identified 
for improvement under NCLB (n-41) 


90 


78 


Identifying and/or developing programs or models that 
address district and/or school needs (n=38) 


95 


79 


Disseminating information on scientifically based research to 
districts and schools (n=36) 


97 


74 


Supporting use of assessment data by schools and districts 
(n^36) 


86 


66 


Formulating or refining state policies to respond to NCLB 
requirements (n=34) 


94 


74 


Providing training and other professional development to local 
educators in academic subjects (reading language arts, 
mathematics, science) (n=29) 


79 


59 


Designing or implementing state assessment and 
accountability systems (n=25) 


80 


65 


Monitoring compliance with NCLB requirements in districts 
and schools (n=24) 


71 


50 


Aligning state accountability systems with NCLB accountability 
systems (n=21) 


81 


62 


Communicating with the public about NCLB requirements or 
report cards (n=16) 


69 


33 



EXHIBIT READS: Of the 46 state managers (weighted) that reported a major or moderate priority for technical 
assistance with building or managing a statewide system of support, 98 percent reported receiving technical 
assistance from the Centers related to that responsibility, and 80 percent reported that the technical assistance was 
major or moderate in scope. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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The intent of the Comprehensive Centers program was that technical assistance from the 
Centers would expand state capacity to carry out important responsibilities under NCLB. 
Although future data collection will look at this in more depth, the survey provided perceptions 
from managers on the extent to which state capacity had been expanded in each area of state 
NCLB responsibility. 

■ Overall, more than two-thirds of state managers (68 percent) reported that 
assistance from the Comprehensive Centers had increased their state’s 
capacity “to a great extent” to carry out its responsibilities in at least one 
NCLB area. On average, state managers reported an improvement in capacity to 
“very great” or “great” extent in 3.6 out of 1 1 areas of NCLB responsibility 
listed on the survey instrument. 

The evaluation looked not just at overall reports of state capacity expansion, but also at 
the capacity building reported in particular areas of NCLB responsibility by those states for 
which the area was a priority. 

■ For each of the four top-ranked areas of priority in NCLB (as described in 
chapter 3), at least two-thirds of state managers for whom that area was a 
technical-assistance priority perceived that Center assistance had expanded 
state capacity to a great or moderate extent. This level of capacity building was 
reported by 67 percent of the managers in building or managing a statewide 
system of support, 71 percent in identifying programs or models for districts or 
schools, 71 percent in training or managing state support teams, and 77 percent 
in disseminating information on scientifically based research. And in at least 49 
percent of the states where any particular NCLB area was a major or moderate 
priority, managers reported great or moderate levels of capacity expansion for 
that area (exhibit 4.4). 



State Uses and Perceptions of the Centers Compared with 
Other Sources of Technicai Assistance 

With many other sources available to states, the Comprehensive Centers were designed to 
fill particular niches — in particular, that of building capacity for state implementation of NCLB 
requirements — rather than addressing every purpose for which states might seek technical 
assistance. The responses of state managers confirmed that the states relied upon multiple 
sources of technical assistance in their practice. 
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Exhibit 4.4. Extent to which Comprehensive Center Assistance expanded state 

capacity for NCLB impiementation, as judged by senior state managers 





Percent of the state managers (weighted) who rated the area 
as a major or moderate technical assistance priority 




Reporting capacity building in this area 


NA, state has 


Area of state responsibility 
under NCLB 


To a very 
great or 
great extent 


To a 

moderate 

extent 


To a small 
or very 
small extent 


Too soon 
to tell 


not sought 
assistance 
for this 
purpose 


Building or managing a statewide system 
of support for districts and schools 
identified for improvement under NCLB 
(n^53) 


53 


14 


18 


8 


7 


Training or managing the state-level staff 
or school support teams who provide 
support to districts and schools identified 
for improvement under NCLB (n-46) 


51 


20 


12 


7 


10 


Identifying and/or developing programs or 
models that address district and/or school 
needs (n-42) 


47 


24 


8 


11 


10 


Formulating or refining state policies to 
respond to NCLB requirements (n-40) 


43 


14 


14 


11 


18 


Supporting use of assessment data by 
schools and districts (n-40) 


42 


16 


11 


2 


29 


Disseminating information on scientifically 
based research to districts and schools 
(n^38) 


39 


38 


10 


5 


8 


Providing training and other professional 
development to local educators in 
academic subjects (reading language arts, 
mathematics, science) (n-38) 


33 


17 


11 


9 


30 


Designing or implementing state 
assessment and accountability systems 
(n=35) 


33 


25 


5 


4 


33 


Aligning state accountability systems with 
NCLB accountability systems (n=28) 


34 


24 


6 


9 


27 


Monitoring compliance with NCLB 
requirements in districts and schools 
(n^25) 


34 


22 


21 


8 


15 


Communicating with the public about 
NCLB requirements or report cards (n=19) 


26 


23 


7 


10 


34 



EXHIBIT READS: Among the 53 state managers (weighted) who reported that technical assistance in building or 
managing a statewide system of support was a major or moderate priority for their state, 53 percent reported that 
technical assistance received from the Comprehensive Centers expanded the state’s capacity in this area to a great 
or very great extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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■ Centers were ranked among the top sources of technical assistance overall 

Thirty-five percent of senior state managers identified the Comprehensive Centers 
as “one of the state’s most important resources,” second to professional 
associations (37 percent) and above the Regional Educational Laboratories 
(33 percent) (exhibit 4.5). 



Exhibit 4.5. States’ use of externai sources of technicai assistance 





Percent of state managers (weighted) reporting: 




External source 


One of the 
state's most 
important 
resources 


To a great 
extent, but 
not one of the 
state's most 
important 
resources 


To a moderate 
extent 


Minimally No contact 


Professional associations (e.g., 
CCSSO, ASCD) (n=54) 


37 


24 


28 


9 


2 


Comprehensive Center network 
(n=54) 


35 


37 


14 


12 


2 


Regional Educational 
Laboratory (n=53) 


33 


29 


21 


12 


5 


U.S. Department of Education 
(n=52) 


29 


16 


35 


19 


1 


Colleges and universities 
(n=54) 


18 


15 


34 


30 


3 


Consulting firms or private 
contractors (n=54) 


13 


29 


33 


19 


6 


Senior managers in other SEAs 
(n=54) 


9 


29 


37 


23 


2 



EXHIBIT READS: Thirty-seven percent of state managers (weighted) said that professional associations were one of 
the state’s most important technical-assistance resources. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 



State managers’ reports of the purposes for which they used each source of technical 
assistance shed light on the niches occupied by the Center program — the purposes for which the 
Centers were the most commonly used source. These responses also revealed some purposes for 
which other sources were the most widely used. 

■ The purposes for which the Centers were the source most often named by the 
state managers were consistent with the Center program ’s purpose, including 
help with start-up and capacity building for state tasks. Using Center 
assistance “to plan the initial steps in solving a problem” was reported as a 
purpose by 66 percent of state managers; using Centers “to develop the skills 
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of SEA or intermediate edueation agency staff’ was reported by 61 percent of 
state managers (exhibit 4.6). Centers were tied with Regional Educational 
Eaboratories as a resource “to complete tasks for which the state lacks 
expertise,” with each of these two sources reported by 54 percent of states as a 
resource for this purpose. 

■ State managers did not cite Centers as a leading resource for three purposes 
that were peripheral to their charge: to gather information or keep current 
with new ideas, to complete tasks that the state could do itself if it had more 
resources, and to work directly with districts and schools. To help states “keep 
current with new ideas,” 96 percent of state managers reported using 
professional associations, 88 percent reported using managers in other SEAs, 
and 87 percent reported using the Centers. Eor “completing tasks the state could 
do itself if it had more staff or resources,” 63 percent reported using consulting 
firms or private contractors, and 59 percent reported using the Centers. Eor 
working with districts and schools, colleges and universities were reported to be 
used by 37 percent of state managers and consulting firms by the same 
percentage; 22 percent reported using the Centers. 
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Exhibit 4.6. Purposes for which states used externai sources of technicai 
assistance 



Purpose in seeking technicai assistance 
(percent of state managers) 



/• 

'h o 












\v 



w\ 



<9 



\ 






% 'o 



^o 



External Source 



Professional associations (e.g., 
CCSSO, ASCD) (n=52) 


96 


51 


Comprehensive Center network 
(n=54) 


87 


66 


Regional Educational Laboratory 
(n=55) 


84 


57 


U.S. Department of Education 
(n=52) 


85 


40 


Colleges and universities (n=54) 


43 


31 


Consulting firms or private 
contractors (n=54) 


30 


35 


Senior managers in other SEAs 
(n=54) 


88 


54 



43 


36 


57 


14 


5 


59 


54 


61 


22 


4 


45 


54 


47 


20 


4 


22 


20 


21 


12 


7 


48 


40 


30 


37 


12 


63 


47 


30 


37 


15 


10 


12 


30 


3 


8 



EXHIBIT READS: Ninety-six percent of state managers (weighted) said that they turned to professional associations 
for the purpose of gathering information or keeping current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 



State managers also provided a rating of the usefulness of Center technieal assistanee, in 
eomparison with other sources, with respect to particular areas of NCLB state responsibility 
(exhibit 4.7). 



■ With regard to the top four areas of NCLB implementation identified by the 

state managers as priorities for technical assistance (see chapter 3), at least half 
of state managers reported that the Centers’ technical assistance was “much 
more” or “somewhat more” useful than that of other providers. When asked to 
compare the usefulness of Comprehensive Center assistance with that of 
assistance from other sources, at least half of state managers reported the Centers 
were more useful in providing assistance in the following areas: building and 
managing statewide systems of support (58 percent of state respondents); 
identifying or developing programs or models for districts or schools (57 percent); 
training or managing the state-level staff or school support teams (55 percent); 
and disseminating information on scientifically based research to districts and 
schools (50 percent). 
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Exhibit 4.7. Usefulness of Comprehensive Center assistance compared with 
assistance from other sources 





Percent of state managers (weighted) rating 
Center technical assistance 


Areas of state responsibility under NCLB 


Much more or 
somewhat 
more useful 


About the 
same 


Much less or 
somewhat 
less useful 


Not able to 
judge 


Building or managing a statewide system of 
support for districts and schools identified for 
improvement under NCLB (n-54) 


58 


16 


14 


12 


Identifying and/or developing programs or models 
that address district and/or school needs (n=48) 


57 


15 


14 


14 


Training or managing the state-level staffer school 
support teams who provide support to districts and 
schools identified for improvement under NCLB 
(n^49) 


55 


17 


11 


17 


Disseminating information on scientifically based 
research to districts and schools (n=51) 


50 


23 


12 


15 


Providing training and other professional 
development to local educators in academic 
subjects (reading language arts, mathematics, 
science) (n=40) 


45 


19 


16 


20 


Formulating or refining state policies to respond to 
NCLB requirements (n=49) 


44 


20 


12 


24 


Supporting use of assessment data by schools 
and districts (n-46) 


37 


21 


21 


21 


Monitoring compliance with NCLB requirements in 
districts and schools (n-39) 


33 


22 


18 


27 


Designing or implementing state assessment and 
accountability systems (n=39) 


29 


22 


22 


27 


Aligning state accountability systems with NCLB 
accountability systems (n=39) 


27 


27 


17 


29 


Communicating with the public about NCLB 
requirements or report cards (n=32) 


22 


26 


15 


37 



NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 
excluded from the analysis. 



EXHIBIT READS: Fifty-eight percent of state managers (weighted) reported that Comprehensive Centers were much 
more or somewhat more useful than other sources of technical assistance for the state responsibility of building or 
managing a statewide system of support. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 

■ For each area of state responsibility in NCLB implementation, there were fewer 
than one-fourth of state managers who reportedly considered the Centers’ work 
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less useful than that of other providers. The areas of NCLB responsibility in 
whieh the largest pereentage of state managers reported that the Centers were less 
useful than other sourees were the following: (1) designing or implementing state 
accountability systems (22 percent of respondents); and (2) supporting use of 
assessment data by schools and districts (21 percent). 



Overall Rating of Center Assistance 

As senior officials in their agencies, the state managers were in a position to provide 
overall judgments of the Center technical assistance their agency had received. The evaluation 
team gathered data on the degree to which Center technical assistance was seen as relevant and 
useful with respect to each of 9 indicators of relevance and 10 indicators of usefulness 
(exhibit 4.8). Ratings were based on a 1 to 5 scale, and the scores for the individual items 
within relevance and within usefulness were averaged for each manager, and for multiple 
managers within a state, as described in chapter 2. 

Exhibit 4.8. Reievance and usefuiness items from the state manager surveys 



Relevance 



Usefulness 



a. Addressed a need or problem that the state 
faces 

b. Addressed an important state priority 

c. Addressed a challenge that the state faces 
related to the implementation of NCLB 

d. Responded flexibly to our state’s changing 
needs 

e. Provided information, advice, and/or resources 
that could be applied to the state’s work 

f. Addressed the particular context in which our 
state operates 

g. Addressed the state’s specific challenges (e.g., 
policy environment, leadership capacity, 
budget pressures, local politics) 

h. Provided information, advice, and/or resources 
that could be used to guide decisions about 
policies, programs, and practices 

i. Highlighted the implications of research 
findings (or information about best practice) for 
state-level clients’ policies, programs, or 
practices 



a. Provided state staff with resources that were 
easy to understand and easy to use 

b. Employed appropriate formats (e.g., work 
groups, conferences, individual consultation, 
written products) 

c. Provided adequate opportunity to learn from 
colleagues in other states 

d. Included adequate follow-up to support the use 
of new information and resources 

e. Were timely 

f. Helped the state to solve a problem 

g. Helped the state to maintain or change a policy 
or practice 

h. Helped the state take the next step in a longer- 
term improvement effort 

i. Provided state staff with information or 
resources that they will use again 

j. Helped state staff to develop skills that they will 
be able to exercise again 



The response options were: 5 (to a very high degree), 4 (to a high degree), 3 (to a moderate degree), 2 (to a low 
degree), and 1 (to a very low degree). 
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■ On a scale of 1 (very low degree) to 5 (very high degree), on average state 

managers gave the Centers’ technical assistance about a 4. The average 
relevanee score was 3.94 and the average usefulness score was 3.86 (exhibit 4.9). 
On both measures, in other words, the mean response was near, but below, the 4.0 
threshold defined as “to a high degree” on the 5-point scale. Fifty-seven percent 
of the states’ ratings fell at or above 4.0 for relevance. Fifty percent fell at or 
above 4.0 for usefulness (exhibit 4.10). Twelve percent of state managers rated 
the relevance and usefulness of the Centers’ work as below moderate. 



Exhibit 4.9. Mean ratings of the reievance and usefuiness of Center assistance, 
as judged by state managers 





Mean 


Mean 




relevance 


usefulness 




rating 


rating 




(n=54) 


(n=53) 


Comprehensive Center 
program 


3.94 


3.86 



EXHIBIT READS: Fifty-four state managers (weighted) gave the Center program a 
mean relevance rating of 3.94. 

SOURCE: Survey of State Managers. Responses weighted so that each state was 
equally represented in instances where more than one manager from a state 
responded. 



Exhibit 4.10: Breakout of ratings of the reievance and usefuiness of Center 
assistance, as Judged by state managers 







Relevance 






Usefulness 






Percent of state managers (weighted) 
giving overall ratings in the following 
ranges: 


Percent of state managers (weighted) 
giving overall ratings in the following 
ranges: 




Very 
low to 
low 

(1.0-1 .9) 


Low to 
moderate 
(2.0-2.9) 


Moderate 
to high 
(3.0-3.9) 


High to 
very 
high 
(4.0-5.0) 


Very low 
to low 
(1.0-1 .9) 


Low to 
moderate 
(2.0-2.9) 


Moderate 
to high 
(3.0-3.9) 


High to 
very 
high 
(4.0-5.0) 


Comprehensive 
Center program 


3 


9 


31 


57 


4 


8 


38 


50 



EXHIBIT READS: Three percent of state managers (weighted) gave the Center program an overall relevance rating 
in the very low to low range (1 .0-1 .9). 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 
where more than one manager from a state responded. 
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5. Variation in the Reported Quaiity, Reievance, and 
Usefuiness of the Centers’ Technicai Assistance 



While state managers’ reports provide important input to an assessment of the overall 
work of the Comprehensive Centers program (ehapter 4), additional perspeetives are helpful in 
assessing the quality of the teehnieal assistanee projeets and their relevanee and usefulness to the 
offiees and the teams of professionals direetly served. A deeper look at variation aeross Centers 
and projeets might also provide information for program improvement: additional support or 
oversight might be provided if quality, relevanee, or usefulness appeared substantially weaker in 
partieular groups of Centers or projeets, or for partieular types of partieipants. Thus, variation 
was investigated by features of projeet design (projeet seope and the type of aetivities offered) 
and by partieipant experienees in the projeet as well as partieipant baekground. 

This ehapter analyzes the ratings of relevanee, usefulness, and quality given to the 122 
Center projeets sampled for the evaluation."^^ Eaeh projeet was evaluated for relevanee and 
usefulness by a sample of partieipants — state staff, intermediate ageney staff, loeal edueators 
working on behalf of the state, and RCC staff — ^who were the intended benefieiaries of the 
projeet and had reeeived at least some of the teehnieal assistanee it provided."^^ Ratings of projeet 
quality were gathered from panels of experts with strong knowledge of the eontent or substantive 
focus of the specific projects they reviewed. Relevance was assessed with eight survey items 
and usefulness with 1 1 items; quality was judged on three items called dimensions (exhibit 5.1). 
Each overall measure (relevance, usefulness, or quality) was calculated as the mean of ratings 
assigned to each item. The item-level ratings themselves were based on 5 -point rating scales 
(exhibit 5.2)."^^ The chapter concludes with a brief discussion of the variation in ratings by 
project characteristics and by the participant experiences in projects. 



See ehapter 2 for more information about the data sourees and proeedures for gathering, ending, and analyzing the data 
reported in this ehapter, ineluding diseussions of the methods used for expert ratings and partieipant surveys. 

Efforts were made to develop parallel wording and mbries that would result in similar gradations between rating 
levels (e.g., very high vs. high vs. moderate) aeross the three measures. However, given the different eontent of eaeh 
set of items within the three measures and the different eontexts for the ratings (experts who underwent training for 
the rating proeess and reviewed identieal paekages of materials vs. survey respondents who typieally partieipated in 
different subsets of projeet aetivities), the ratings aeross the three measures are not direetly eomparable. 
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Exhibit 5.1. Quaiity, reievance, and usefuiness items 



From expert panel scoring From project participant surveys 

Technical quality Relevance Usefulness 



Reviewers were directed to 
assign a score to each 
dimension and to include the 
basis for their ratings on the 
rating form, including the specific 
artifacts on which their score was 
based. The three dimensions 
are: 

a. Demonstrated use of the 
appropriate documented 
knowledge base-to include 
an accurate portrayal of the 
current state of information 
with prominence to those with 
the most accurate/rigorous 
evidence 

b. Fidelity of application of the 
knowledge base to the 
products and services 
provided-materials are 
consistent with the 
best/accurate information 
available and the presentation 
adequately conveys the 
confidence of the information 

c. Clear and effective delivery- 
information is well organized 
and written and accessible to 
the intended audience for 
easy use 



Based on your experience, to 
what degree was this set of 
activities and resources relevant 
to your work, in each of the 
following respects? 

a. Addressed a need or problem 
that my organization faces 

b. Addressed an important 
priority of my organization 

c. Addressed a challenge that 
my organization faces related 
to the implementation of NCLB 

d. Provided information, advice, 
and/or resources that could be 
directly applied to my 
organization’s work 

e. Addressed our particular state 
context 

f. Addressed my organization’s 
specific challenges (e.g., 
policy environment, leadership 
capacity, budget pressures, 
local politics) 

g. Provided information, advice, 
and/or resources that could be 
used to guide decisions about 
policies, programs, or 
practices 

h. Highlighted the implications of 
research findings (or 
information about best 
practice) for policies, 
programs, or practices 



Based on your experience, to 
what degree was this set of 
activities and resources useful to 
you, in each of the following 
respects? 

a. Provided resources that were 
easy to understand and easy 
to use 

b. Employed an appropriate 
format (e.g., a work group, a 
conference, individual 
consultation, written products) 

c. Provided adequate opportunity 
to learn from colleagues in 
other states 

d. Included adequate follow-up to 
support the use of new 
information and resources 

e. Were timely 

f. Helped my organization solve 
a problem 

g. Helped my organization 
maintain or change a policy or 
practice 

h. Helped my organization take 
the next step in a longer-term 
improvement effort 

i. Provided my organization with 
information or resources that 
we will use again 

j. Helped my organization 
develop a shared expertise or 
knowledge base 

k. Helped individuals in my 
organization to develop skills 
that they will use again 
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Exhibit 5.2. Scaies for rating of quaiity by expert paneis and reievance 
and usefuiness by participants 



Rating 


Quality ratings by 
content area experts 


Relevance or usefulness 
ratings by project 
participants 


5 


Very high — All or almost 
all of the project meets the 
given indicators for a 
dimension 


To a very high degree 


4 


High — Most of the project 
meets the given indicators 
for a dimension 


To a high degree 


3 


Moderate — Some of the 
project meets the given 
indicators for a dimension 


To a moderate degree 


2 


Low — Limited parts of the 
project meet the given 
indicators for a dimension 


To a low degree 


1 


Very low — None or 
almost none of the project 
meets the given indicators 
for a dimension 


To a very low degree 



Based on these ratings, the key findings suggest: 

■ Center technical assistance was judged in the “moderate” to “high ” range of 
quality, relevance, and usefulness. On a seale of 1 to 5 with a 3 representing 
“moderate” and a 4 representing “high,” the programwide average ratings for the 
sampled projeets were 3.34 for teehnieal quality (seored by panels of eontent 
experts), and 3.94 for relevanee and 3.70 for usefulness (seored by partieipants).^° 

■ The average quality rating was higher among CCs than RCCs; relevance and 
usefulness ratings were similar between the two Center types. The Content 
Centers reeeived Center-level mean seores for teehnieal quality that averaged 
3.73, eompared with 3.21 for the Regional Centers; the differenee of 0.52 points 
exeeeded one-half of one pooled standard deviation. The seores for relevanee and 



This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 

Using Cohen (1988) as a conceptual framework, we estimated Cohen's d (an estimate of the effect size defined as 
the differences in means divided by the pooled standard deviation) and adopted the logic of Cohen for what is a 
moderate difference. Specifically, we adopted a difference in the means of one-half of one standard deviation 
(analogous to an effect size of .5) as our minimum threshold for highlighting differences. The “pooled standard 
deviation” for each computation varied with the unit of analysis. For analyses conducted at the Center level, the 
pooled standard deviation was computed as the standard deviation of the variable of interest (e.g., relevance) 
computed at the Center level. For analyses using the project as the unit of analysis, the pooled standard deviation 
was computed at the project level. 
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usefulness were 3.99 and 3.71 for the Regional Centers and 3.78 and 3.65 for the 
Content Centers; on these measures the two types of Centers seored within one- 
half of a standard deviation of eaeh other. 

■ Mean ratings of quality, relevance, and usefulness varied across and within 
Centers. On eaeh measure, at least 10 Centers had a mean rating that was at least 
one-half of a standard deviation above or below the overall mean for its type of 
Center (RCC or CC) for that measure (i.e., 11 of 21 Centers were this far above or 
below the mean for quality, 10 for relevanee, and 14 for usefulness). However, 
few Centers showed eonsistently high or low ratings aeross measures: one RCC 
had ratings of quality, relevanee, and usefulness for its sampled projects that 
exceeded the RCC means for each measure by at least one-half of a standard 
deviation; one RCC and one CC had ratings at least one-half of a standard 
deviation lower than those of all RCCs and CCs, respectively, on all three 
measures. Ratings varied across projects within Centers: at least one project was 
rated “high” (4.0 or above) for quality in 1 1 of the 16 RCCs and 3 of the 5 CCs; 
the same was true for relevance in 15 of the 16 RCCs and all 5 CCs; and for 
usefulness in 1 1 of the 16 RCCs and all 5 CCs. 

■ Across subgroups of projects, average project ratings differed by project scope 
but generally did not differ by the presence of particular types of activities. 

Projects that Centers had designated as large in scope (“major” projects), relative 
to the rest of that Center’s own work, had relevance and usefulness ratings at least 
one-half of a standard deviation higher than those for the moderate projects 
sampled (exhibit 5.7). With respect to project activities, the differences in ratings 
for projects that either did or did not offer each of five types of Center activities or 
resources (research syntheses, training, task force participation, engagement of 
participants in planning, or development of a formal implementation plan) were 
less than one -half of a standard deviation (see exhibit 5.8). Those projects that 
offered ongoing consultation or support among their activities had lower quality 
ratings than did other projects, but higher relevance ratings; quality was rated 
higher for projects that included conferences among their activities, compared 
with projects that did not offer conferences. Each of the differences highlighted 
was at least one-half of a standard deviation in size. 

■ Ratings of quality were not statistically related to ratings of relevance and 
usefulness. The correlations between the scores for quality and relevance (-0.12) and 
between the scores for quality and usefulness (-0.04) are not statistically significant, 
meaning we cannot be sure that they are different from zero (no relationship). This 
indicates that the expert rating of technical quality was not related to the extent to 
which participants deemed the projects to be relevant or useful. 

■ Participants who reported more active and extensive experience with a project 
gave higher ratings for relevance and usefulness than did other participants. 
Because participant experiences could and did vary within projects as well as 
across projects, the descriptive comparisons here are at the individual participant 
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level. Higher ratings were found among partieipants who were involved upfront in 
projeet design (eompared with those who were not), who spent at least three days 
in any projeet aetivity (eompared with those who partieipated for fewer days in 
that aetivity), and who spent at least three days in eaeh of a higher number of 
project activities (compared with those who spent that much time in fewer 
activities). These differences were statistically significant at the .05 level. These 
descriptive analyses merely show differences among participants, and should not 
be interpreted as implying that active or extensive experience with a project led to 
higher ratings. 



Overall Ratings of Center Projects by Participants and Experts 

The data gathered on individual projects were compiled to produce mean ratings of 
quality, relevance, and usefulness for the sampled projects across the program as a whole. In 
these averages, each Center contributed equally to the overall mean for the program. Given the 
different tasks that the RCCs and CCs were charged with, it is appropriate to examine whether 
there were differences in ratings between the two types of Centers. In addition, the overall 
means for the two Center types potentially mask variation in the quality, relevance, and 
usefulness across individual Centers that would be useful to understand. 

■ Averaged across the Centers, the quality, relevance, and usefulness of sampled 
projects were judged in the “moderate” to “high” range. Overall, the mean 
ratings were 3.34 for technical quality (scored by the panels of content experts), 
and 3.94 for relevance and 3.70 for usefulness (as scored by participants in the 
sampled projects) (exhibit 5.3). The rating scales established a 3 as indicating a 
moderate degree of quality, relevance, and usefulness, while a score of 4 
represented a high degree of relevance and usefulness (see exhibit 5.2). The 
average of the ratings for each measure fell between a score of 3 and 4. 

■ Compared with Content Centers, Regional Centers received a lower score from 
reviewers on technical quality, but a higher score from participants on 
relevance, while the usefulness score was similar between the two Center types. 
There was a gap of more than one -half of a standard deviation between the two 
types of Centers on technical quality of -0.52, with the CCs receiving an average 
score of 3.73 and the RCCs receiving an average score of 3.21 . The difference in 
the relevance score (0.21) exceeded one-half standard deviation, while there was 
no substantive difference in usefulness ratings. 



ED structured the Comprehensive Centers program to consist of 5 Content Centers charged with specializing in 
activities related to specific content areas and 16 Regional Centers charged with specializing in interactions with the 
states in a region. 

For analyses conducted at the Center level, the pooled standard deviation was computed as the standard deviation 
of the variable of interest (e.g., relevance) computed at the Center level. 
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Exhibit 5.3. Center Levei Mean ratings of technicai quaiity, reievance, and 
usefuiness 



Quality dimensions: 





Technical 

quality 


Dimension 1 
(use of 
knowledge 
base) 


Dimension 2 

(fidelity of 
application) 


Dimension 3 
(clear and 
effective 
delivery) 


Relevance 


Usefulness 


All 


Comprehensive 
Centers (N=21) 


3.34 


3.22 


3.20 


3.60 


3.94 


3.70 


All RCCc 


(N=16) 


3.21 


3.05 


3.07 


3.52 


3.99 


3.71 


All CCs (N=5) 


3.73 


3.75 


3.59 


3.85 


3.78 


3.65 


Difference of 


RCC and CC 
means 


-0.52^ 


-0.70^ 


-0.52^ 


-0.33^ 


0.2V 


0.06 


Pooled standard 


deviation(all 

Comprehensive 

Centers) 

Ratio of difference 
in means to 


0.41 


0.49 


0.44 


0.35 


0.34 


0.34 


pooled standard 
deviation 


- 1.28 


- 1.42 


- 1.18 


- 0.95 


0.60 


0.18 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The 


“technical quality” rating is the mean of the 


ratings for the three quality dimensions. A notation of ' 


indicates that the difference in the mean ratings between the 



CCs and RCCs is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for relevance 
and usefulness. Responses weighted so that each panelist or participant contributed equally to project ratings; each 
project contributed equally to Center ratings; and each Center contributed equally to cross-Center ratings. 

■ There was variation in the ratings across and within individual Centers. On 

each measure, at least 1 1 Centers had a mean rating that was at least one -half of a 
standard deviation above or below the overall mean for its type of Center^"^ (RCC 
or CC) for that measure (i.e., 11 of 21 Centers were this far above or below the 
mean for quality, 1 1 for relevance, and 14 for usefulness). One RCC was rated 
higher than others by at least one -half of a standard deviation on all three 
measures, and one CC and one RCC were rated lower than others on all three 
measures. Aside from these Centers, the other 18 Centers’ ratings were not 
consistently higher or lower than the mean but varied across measures (exhibit 
5.4). 



For analyses using the projeet as the unit of analysis, the pooled standard deviation was computed at the project 
level. 
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Exhibit 5.4. Mean ratings of technicai quaiity, reievance, and usefuiness, 
by Center 



Center type 


Technical 

quality 


Relevance 


Usefulness 


RCCs (N=16) 


3.78t 


3.784 


3.424 




3.63t 


3.224 


3.004 




3.46t 


4.18t 


3.94t 




3.44t 


3.90 


3.63 




3.36 


3.97 


3.63 




3.35 


3.97 


3.57 




3.35 


3.93 


3.514 




3.31 


4.15 


3.69 




3.21 


4.08 


3.82 




3.17 


4.3lt 


4.05t 




3.15 


4.12 


3.93t 




3.11 


4.70t 


4.46t 




2.984 


4.07 


3.92t 




2.744 


3.204 


3.054 




2.744 


4.01 


3.54 




2.634 


4.18t 


4.17t 


Average RCC rating 


3.21 


3.99 


3.71 


Pooled standard 


deviation (RCCs) 


U.32 


U.-if 


U.33 


CCS (N=5) 


4.24t 


3.76 


3.544 




3.94 


3.90t 


3.86t 




3.88 


3.99t 


3.84t 




3.444 


3.584 


3.444 




3.144 


3.684 


3.56 


Average CC rating 


3.73 


3.78 


3.65 


Pooled standard ... ... ... 


deviation (CCs) 


U.4J 


0.7 b 


0.79 



NOTE: The arrow pointing upward indicates the accompanying value is at least one-half of one standard 
deviation above the group mean (e.g., 3.78 is at least one-half of one standard deviation above the mean for 
the RCCs). The arrow pointing downward indicates the accompanying value was at least one-half of one 
standard deviation below the group mean. 

EXHIBIT READS: One of the RCCs had a mean rating for technical quality of 3.78, a mean rating for 
relevance of 3.78, and a mean rating for usefulness of 3.42, across the projects sampled from that Center. 
SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to 
project ratings, and each project contributed equally to Center ratings 
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Variation by Type of Project 



Perhaps the Centers were more sueeessful, at least from the perspeetive of the reviewers, 
at some types of projeets eompared with others. Thus, the evaluation looked at different groups 
of projeets, aeross Centers, that might be expeeted to show higher quality, relevanee, or 
usefulness than other projeets. For example, projeets that were larger in seope than others in the 
Center might have garnered higher ratings from the experts or the partieipants. Projeets that 
offered partieular types of aetivities or resourees might also stand out for higher ratings. 

These deseriptive subgroup analyses were possible beeause there was variation in ratings 
among the individual projeets within the sample (exhibit 5.5). On teehnieal quality, 4 pereent of 
the RCC and CC projeets were judged “very low to low,” 29 pereent and 4 pereent of RCC and 
CC projeets respeetively were seored in the “low to moderate” range, 50 and 52 pereent seored 
in the “moderate to high” range, and 17 pereent and 41 pereent seored in the “high to very high” 
range. The ratings for relevanee and usefulness also varied aeross projeets. 



Exhibit 5.5. Distribution of technicai quaiity, reievance, and usefuiness 
ratings - percent of projects 

Technical quality 

Percent of projects 

100 T , 



80 




Very low to low Low to moderate Moderate to high High to very high 
(1.0-1.9) (2.0-2.9) (3.0-3.9) (4.0-5.0) 



Relevance 



Usefulness 



Percent of projects 
100 

80 

60 

40 

20 

0 

Very low to low Low to moderate Moderate to high High to very high 
(1.0-1 .9) (2.0-2.9) (3.0-3.9) (4.0-5.0) 



57 




Percent of projects 
100 

80 

60 

40 

20 

0 

Very low to low Low to moderate Moderate to high High to very high 
(1.0-1 .9) (2.0-2.9) (3.0-3.9) (4.0-5.0) 
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Key: S RCCs □ CCs 

EXHIBIT READS: For technical quality, 4 percent of the projects sampled from RCCs had a mean quality rating of 
very low to low (between 1.0 and 1 .9 on a 5-point scale). 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating. 
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These distributions of ratings aeross projeets were wider than the distributions aeross 
Centers, whieh on all three measures were elustered in the “moderate to high” range as deseribed 
above. At least one project was rated “high” (at least 4.0) for quality in 1 1 of the 16 RCCs and 3 
of the 5 CCs; the same was true for relevance in 15 of the 16 RCCs and all 5 CCs; and for 
usefulness in 1 1 of the 16 RCCs and all 5 CCs. Among RCCs, four or fewer had at least one 
project rated “low” (2.0 or lower) on quality, relevance or usefulness. One CC had at least one 
project rated “low” for quality, but none of the CC projects received a “low” rating for relevance 
or usefulness (exhibit 5.6). 



Exhibit 5.6. Technicai quaiity, reievance, and usefuiness ratings, by projects 
rated “high” and projects rated “iow” and Center type 





Technical 

quality 


Relevance 


Usefulness 


Regional Comprehensive Centers (RCCs) (N=16) 






RCCs with at least one project 
rated “high” (at least 4.0) 


11 


15 


11 


RCCs with at least one project 
rated “low” (2.0 or less) 


4 


3 


3 


Content Centers (CCs) (N=5) 








CCs with at least one projected 
rated “high” (at least 4.0) 


3 


5 


5 


CCs with at least one project 
rated “low” (2.0 or less) 


1 


0 


0 



EXHIBIT READS: For technical quality, 11 of the 16 RCCs had at least one project that received a “high” 
rating of at least 4.0. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants 
for relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to 
the project rating. 



Ratings by Project Size or Levei of Effort 

The quality or utility of projects might be related to the scale of the undertaking. For 
example, more ambitious projects that extended over a longer period of time or included a larger 
number of participants might have been more challenging to carry out but might have been 
conceived to address a great need. Larger projects might allow greater focus on the effort and 
enjoy relatively higher levels of Center resources. Alternatively, the tighter focus possible in a 
smaller-scale project might have been conducive to higher quality, relevance, or usefulness. 
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The Centers were asked to consider the scope of each of the 122 sampled projects (i.e., its 
cost or level of effort) in relation to other projects of that Center and to designate each project as 
either “major” or “moderate.” 

■ Projects identified by the Centers as “major” efforts were rated higher on the 
measures of relevance and usefulness. Participants gave the major projects a 
mean rating of 4.02 for relevance, which was one-half of a standard deviation 
higher than the mean rating of 3.76 that they gave the moderate projects 
(exhibit 5.7). The mean usefulness rating for the major projects was 3.77, 
exceeding the corresponding rating for the moderate projects (3.47) by more than 
one-half of a standard deviation. 

■ The quality ratings for major and moderate projects were within one-half of a 
standard deviation of each other, however. The means were 3.37 and 3.25 
respectively, both close to the “moderate quality” rating of 3. 



Exhibit 5.7. Technicai quaiity, reievance, and usefuiness ratings, by project size 



Project size 


Technical 

quality 


Relevance 


Usefulness 


Major projects 


3.37 


4.02 


3.77 


Moderate projects 


3.25 


3.76 


3.47 


Difference in means 


0.12 


0.26^ 


0.30^ 


Pooled standard deviation 


0.69 


0.52 


0.54 


Ratio of difference in means 
to pooled standard deviation 


0.18 


0.50 


0.55 



NOTE: For technical quality the overall number of projects was 122 with 93 major projects and 29 moderate projects. 
For relevance and usefulness the overall number of projects was 120, with 91 major projects and 29 moderate 
projects. 

EXFIIBIT READS: The mean technical quality rating for major projects in the sample was 3.37 while the mean 
technical quality rating for moderate projects was 3.25. The ^ indicates that the difference in the mean ratings 
between the major and moderate projects exceeds one-half of the pooled standard deviation in the rating. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 



Ratings by Project Activities 

The technical assistance provided by the Centers varied in delivery methods, including the 
types of activities offered. We examined whether the inclusion of certain types of technical 
assistance activities resulted in more favorable project ratings either by experts knowledgeable 
about the topic or by participants in the project (exhibit 5.8). The types of activities implemented by 
the Centers and coded by the evaluation team (first introduced in chapter 3) overlapped, in that 
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several eould be ineluded in a single project. By comparing the ratings of projects that offered 
each activity with ratings of projects that did not, it may be possible to identify hypotheses for 
further exploration about strengths and weaknesses of particular modes of Center work. 



■ Across five of the seven project activities listed here, comparisons of projects 
with and without the activities showed no differences in quality, relevance, or 
usefulness ratings greater than one-half of a standard deviation (exhibit 5.8). 
Those projects that offered ongoing consultation among their activities had lower 
quality ratings than other projects, but higher relevance ratings; quality was rated 
higher for projects that included conferences among their activities, compared 
with projects that did not offer conferences. Each of the differences highlighted 
was at least one-half of a standard deviation in size. All other differences in 
exhibit 5.8 were within one-half of a standard deviation. 



For each sampled project, the Center furnished the team with a cover sheet intended to inform the expert 
reviewers about the project context, purpose, activities, and products. As described in chapter 2 and appendix C, the 
team coded the content of each project cover sheet to describe the activities and resources offered as part of the 
project. When a cover sheet indicated that the project had multiple components (which was the case for 84 percent 
of the projects), multiple codes were assigned. 
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Exhibit 5.8. Technicai quaiity, reievance, and usefuiness ratings of projects, by 
type of activity offered in the projects 



Type of assistance activity offered 


Technical quality Relevance Usefulness 


Ongoing Yes 

consultation or 
support 


3 

1 

3 

1 


;.25 4.03 3.74 

(n=84) (n=82) (n^82) 

;.59 3.79 3.60 

(n=38) (n=38) (n=38) 


□iff 


SD 


Ratio 


-0.34^ 


0.50 


-0.68 0.24'*' 


0.46 


-0.52 0.14 


0.54 


0.26 


Research Yes 

collections and 
syntheses 


3.50 3.94 3.67 

(n=71) (n=70) (n=70) 

3.17 3.98 3.73 

(n=51) (n=50) (n^50) 


□iff 


SD 


Ratio 


0.33 0.67 0.49 -0.04 0.52 -0.08 -0.06 0.54 -0.11 


Engagement of Yes 

participants in 
project planning 


3 

I 

3 

I 


;.24 4 

(n=50) 

;.44 3 

(n=m I 


.00 3.73 

(n=48) (n=48) 

.93 3.67 

(n=m (n=m 


□iff 


SD 


Ratio 


-0.20 


0.67 


-0.30 0.07 


0.52 


0.14 0.06 


0.54 


0.11 


Training events Yes 

No 


3 

I 

3 

I 


;.34 4.09 3.83 

(n=50) (n^49) (n^49) 

;.37 3.86 3.60 

(n=72) (n=71) (n=71) 


□iff 


SD 


Ratio 


-0.03 0.67 -0.05 0.23 0.52 0.44 0.23 0.54 0.43 


Taskforce Yes 

meetings and work 

No 


3 

I 

3 

I 


;.23 4.05 3.71 

(n=50) (n^48) (n^48 

;.45 3.89 3.68 

(n=72) (n=72) (n=72 


') 


□iff 


SD 


Ratio 


-0.22 


0.52 -0.42 0.16 0.54 0.30 0.03 0.67 


0.04 


Conference Yes 

No 


3.60 3.83 3.61 

(n^43) (n^43) (n^43) 

3.21 4.03 3.74 

(n=79) (n=77) (n=77) 


□iff 


SD 


Ratio 


0.39''^ 


0.67 


0.58 -0.20 


0.52 


-0.40 -0.13 


0.54 


-0.24 


Support Yes 

development of a 
formal plan to 
implement a 


3.38 4.02 3.63 

(n=20) (n=20) (n=20) 

3.35 3.94 3.71 

(n=102) (n=100) (n=100) 


program or policy Diff 


SD 


Ratio 


0.03 


0.67 


0.04 0.08 


0.54 


0.15 -0.08 


0.54 


-0.15 



NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 
sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 
means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 
deviation. A notation of t indicates that the difference in the mean ratings is at least one-half of one pooled standard 
deviation. 

EXHIBIT READS: The mean quality rating of projects that offered ongoing consultation or support was 3.25, while the 
mean quality rating of projects that did not offer this was 3.59. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 
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Ratings for RCC Projects by Whether They Drew on CC Contributions 

Cooperation between CCs and RCCs was one element of the design of the 
Comprehensive Centers system, as discussed throughout this report. Thus, among the types of 
resources that RCC projects might offer to their participants were materials developed by CCs, 
activities developed with advice from CCs, or technical assistance from CC staff members. As 
indicated in chapter 3, 46 of the 96 sampled RCC projects had one or more of these types of CC 
contributions; the other 50 did not. One hypothesis for the evaluation was that the RCC projects 
incorporating CC contributions would be rated higher for technical quality than RCC projects 
that were developed without CC input, since the CCs were specifically charged with synthesizing 
the knowledge base in their areas of focus, and since the projects sampled from CCs had 
received a higher mean rating of technical quality than those sampled from RCCs. 

■ Among the RCC projects sampled, the ratings of those with CC contributions 

did not differ by more than one-half of a standard deviation on any measure, 
compared with those without CC contributions (exhibit 5.9). For example, the 
RCC projects with CC contributions had a mean quality rating of 3.40, which was 
just under one-half of a standard deviation (0.47) of the rating of those that did not 
include CC contributions (3.09). 



Exhibit 5.9. Technicai quaiity, reievance, and usefuiness ratings by Content 
Center contribution to RCC projects 



RCC projects with... 


Technicai 

quaiity 


Reievance 


Usefuiness 


CC contribution (n=46) 


3.40 


3.92 


3.63 


No CC contribution (n=50) 


3.09 


4.09 


3.78 


Difference in means (with CC 
contribution vs. without) 


0.31 


-0.17 


-0.15 


Pooled standard deviation 


0.67 


0.53 


0.56 


Ratio of difference in means to pooled 
standard deviation 


0.47 


-0.32 


0.27 



NOTE: For the technical quality ratings, there were 46 RCC projects reporting a contribution from one or more CC 
while 50 RCC projects reported no CC contribution. For the relevance and usefulness ratings there were 45 RCC 
projects with a CC contribution and 49 projects with no CC contribution. 

EXFIIBIT READS: The mean technical quality rating for RCC projects with a CC contribution was 3.40 while the mean 
technical quality rating for RCC projects with no CC contribution was 3.09. 

SCURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 
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Relationships between the Relevance, Usefulness, and 
Quality of Center Projects 



In our analysis of the relevance, usefulness, and quality of Center technical assistance, we 
observed that some groups of projects that received relatively high ratings for quality received 
relatively low ratings for relevance and usefulness, and vice versa. We hypothesized that 
differences in the rating and rankings (the rating relative to other groups of projects) could be 
due to the differences in the types of individuals who reviewed and scored projects for quality — 
content experts — and those who scored the relevance and usefulness — project participants. It was 
expected that these two groups might value and be better able to judge different qualities in a 
Center project, which is why we did not have content experts evaluate the projects for their 
utility or the participants assess the technical quality. 

We examined the associations among the three dimensions more systematically by 
calculating correlation coefficients.^^ These statistics indicate the strength and direction of a linear 
relationship between two factors. A correlation coefficient can vary from positive one (indicating a 
perfect positive relationship), through zero (indicating the absence of a relationship), to negative one 
(indicating a perfect negative relationship). If the correlation is statistically significant (p <.05), we 
can have strong (95 percent) confidence that what we calculated is not due to chance. 

■ Participants’ ratings of relevance and usefulness were closely related 
(exhibit 5.10). The correlation coefficient is +.84. This indicates that the extent to 
which participants rated the projects as relevant was associated with how they 
deemed the project to be useful to their agency. 

■ There was no statistically significant relationship between the ratings for the 
quality of Center projects and for their relevance or usefulness (exhibit 5.10). 

The results indicate correlations of -0.12 between quality and relevance, and -0.04 
between quality and usefulness. Although the coefficients are negative, because 
they are not statistically significant we cannot be sure that they are different from 
zero (no relationship). Still, the correlations suggest that the expert rating of 
technical quality was not related to the extent to which participants deemed the 
projects to be relevant or useful. This was also the case for the dimension-level 
ratings within the overall quality rating: none was related to the rating of relevance 
or usefulness. This latter finding will be explored further in the final evaluation 
report. 



For this analysis, the evaluation team used Spearman’s rank order eorrelation beeause this non-parametrie rating is 
the appropriate statistieal funetion to deseribe eorrelations between two variables where the values of the variables 
are not normally distributed and are on a seale (sueh as ratings). 
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Exhibit 5.10. Correiations between project-ievei technicai quaiity, reievance, 
and usefuiness ratings 



Combination 


Spearman’s rho 


Quality and relevance 


-0.12 


Quality and usefulness 


-0.04 


Relevance and usefulness 


+0.84 



EXHIBIT READS: The correlation between project-level ratings of technical 
quality and relevance ratings was a negative 0.12. 

SOURCE: Expert panel ratings of sampled projects for technical quality and 
surveys of project participants for relevance and usefulness. Responses 
weighted so that each panelist or participant contributed equally to the project 
rating, and each project contributed equally to the mean rating for its category. 



Variation by Participant Experiences and Roies 

It is possible that Center technieal assistanee is viewed as more benefieial by some types 
of partieipants or that the extent to whieh partieipants are engaged in the projeets affeets how 
they regard it. These questions ean be analyzed by examining whether the rating of relevanee or 
usefulness given by partieipants was related to a partieipant’s experienee in the project or to his 
or her job responsibilities.^^ These analyses provided an opportunity to follow up on findings 
from the prior evaluation conducted on the Comprehensive Regional Assistance Centers 
established under the 1994 ESEA reauthorization (described in chapter 1 of this report). The 
specific participant characteristics examined included; 



■ Whether the participant contributed to the design of the project 

■ How much time the participant spent with project activities or resources 

■ The participant’s role in his or her job, and the extent to which the job focused on 
NCLB 

In the following discussion, the unit of analysis is the participant, not the project. The 
mean ratings are those provided by a group of participants who had a particular type of 
experience or background; within the same project there could be and were participants with 
different experiences and backgrounds. In addition, participants could and did participate in quite 
different subsets of all activities offered in a project. A typical project comprised multiple types 



Within this section, the counts in the tables represent the weighted number of project participants who responded 
to that item. Because respondents were sampled at random from complete lists of participants in each sampled 
project, it was possible to draw inferences from their responses to the universe of participants in the 120 projects 
rated for relevance and usefulness. Accordingly, this section reports the results of tests of statistical significance. 

U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 
Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 
Evaluation (Volume I). Washington, DC: Author. 
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of activities. The median number of aetivities was 4.0, and ranged from 1 to 7. Thirty-two 
pereent of partieipants took part in three or more aetivities, suggesting that it was unusual for the 
same individuals to partieipate in all of a projeet’s aetivities. Just over half of projeet partieipants 
(52 pereent) reported eontributing to the design of the projeet. In the following analysis, we 
examine whether different patterns of partieipation in a projeet were assoeiated with different 
ratings of the projeet’s relevanee and usefulness. 



Ratings by Participants’ Invoivement in Project Design 

An aspeet of partieipation that was of partieular interest, given the Centers’ eharge to work 
elosely with their elients, was the extent to whieh partieipants were involved in the projeets’ design 
stages and, then, whether pereeptions of relevanee and usefulness might vary with a partieipant’s 
involvement. The evaluation of the earlier program of Comprehensive Regional Assistanee 
Centers, based on survey results from a sample of partieipants, eoneluded: “Comprehensive Center 
teehnieal assistanee is more useful to customers if it is tailored to address their needs and 
interests. For the eurrent evaluation, the team analyzed partieipant responses to a question 
asking respondents whether they were personally involved in determining the goals or designing 
the eontent or format of the projeet. 

■ A higher proportion of participants in the sampled RCC projects reported being 
involved in some aspect of determining project goals or designing the project 
than did participants in the CC projects. The figures were 57 pereent for 
partieipants in RCC projeets and 42 pereent for those in CC projeets 

(exhibit 5.1 1). A chi square test found this differenee to be statistieally signifieant 

(p<.01).'’° 

■ Among participants in the sampled Content Center projects, RCC staff were 
more likely to be involved in some aspect of determining project goals or 
designing the project than were state-level staff (exhibit 5.12). Forty-five pereent 
of RCC staff who partieipated in teehnieal assistanee provided by the CCs were 
involved in the projeet’ s design. The eorresponding figure was 36 pereent among 
“state-level” staff (a eategory that ineluded not only SEA employees but also loeal 
edueators and others working with a state team or task foree (as deseribed in 
ehapter 2). A ehi square test found this differenee in proportions to be statistieally 
signifieant (p<.01).^' 



U.S. Department of Education, 2000, op cit. 

“ Unlike the analyses at the project level, which are based on a purposive sample of projects and thus could not use 
the tools of inferential statistics, these analyses at the participant level are based on random sampling of participants; 
thus they can be generalized to all participants in the sampled projects using conventional statistical tests. 

No parallel analysis could be conducted for CC participants in RCC projects because there were none; the RCCs 
did not provide technical assistance to CC staff 
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Exhibit 5.11. Participants’ invoivement in determining project goais or design, 
by type of Center 



Involved in determining 
the goals or design of 
the project 


Percent of participants 


RCC projects 

(n=2,355) 


CC projects 

(n=1,116) 


Total 


100 


100 


Yes 


57 


42 


No 


43 


58 



NOTE: Difference in the proportion of participants involved in design by type of Center is 
statistically significant (p<.01, Chi Square). 



EXHIBIT READS: Fifty-seven percent of participants in projects conducted by RCCs reported 
they were involved in determining the goals or design of the project. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented 
all sample-eligible participants for the project. 



Exhibit 5.12. Participants’ invoivement in determining project goais or design, 
by type of participant organization 



Participant involvement 


Percent of 
state-level 
staff 

(n^455) 


Percent of 
RCC staff 

(n=661) 


Total 


100 


100 


Involved in determining the goals or 
designing the content of the CC project 


36 


45 


Not involved in determining the goals or 
designing the content of the CC project 


64 


55 



NOTE: Difference in the proportion of participants involved in design by type of participant 
organization is statistically significant (p=.01, Chi Square). 

EXHIBIT READS: Thirty-six percent of the state-level staff who participated in Content Center 
projects reported involvement in determining the project’s goals or design. 

Source: Surveys of project participants. Responses weighted so that respondents represented all 
sample-eligible participants for the project. 



■ Participants who reported being involved in determining the goals or designing 
the content of the project provided higher relevance and usefulness ratings than 
did other participants. The average ratings in projeets sampled for the eurrent 
study eorroborate the survey findings from the prior evaluation of the Centers; 
partieipants who reported involvement in projeet design gave the projeets a 4.06 
average seore for relevanee and a 3.83 score for usefulness (exhibit 5.13). In 
contrast, average scores for participants not involved in project design were 3.81 
and 3.56, respectively. These differences were statistically significant (with p<.01 
for differences in both relevance and usefulness). This finding stands in contrast 
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to the one displayed in exhibit 5.8, which showed no difference in ratings among 
projects that offered some opportunity for involvement compared with projects 
that did not do so. This suggests that involving some participants in design work 
may make a difference in the ratings provided by those particular participants, 
although not in the ratings provided by all participants in the project. 



Exhibit 5.13. Mean reievance and usefuiness ratings, by respondent invoivement 
in determining project goais or design 



Respondent 

involvement 


Relevance 


Usefulness 


Involved in project 
design (n=1,802) 


4.06 


3.84 


Not involved in project 
design (n=1,638) 


3.81 


3.56 


Difference 


0.25* 


0.28* 


Significance 


p<.01 


p<.01 



* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: Participants who indicated that they were involved in determining the 
goals or design of the project provided a relevance rating of 4.06, which was statistically 
significantly higher than the relevance rating of 3.81 provided by participants who indicated 
that they were not involved in determining the goals or design of the project. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 
represented all sample-eligible participants for the project. 



Ratings by Time Spent in Project Activities 

The literature on professional development suggests that the duration of training is 

62 

associated with better outcomes. Similarly, the previous evaluation of the Comprehensive 
Centers program, based on survey data from participants, concluded: “Comprehensive Center 

63 

technical assistance is more useful to customers if it is intensive and if it extends over time.” 
Therefore, this study explored the possibility of a relationship between the time an individual 
spent in one or more project activities and the ratings he or she provided, assessing whether the 
same relationship persisted. 



Desimone, L., Porter, A.C., Caret, M., Yoon, K.S., and Birman, B. (2002). “Does Professional Development 
Change Teachers’ Instmction? Results from a Three-Year Study.” Educational Evaluation and Policy Analysis, 
24(2), 81-112; Caret, M., Porter, A., Desimone, L., Birman, B., and Yoon, K.S. (2001). “What Makes Professional 
Development Effective? Results from a National Sample of Teachers.” American Educational Research Journal, 
38(4), 915-945. 

“U.S. Department of Education, 2000, op cit., p. 9. 

At the participant level, the variable measured in the survey was the amount of time an individual respondent 
reported devoting to each of eight activities in which a participant might engage. For each activity, respondents 
indicated the amount of time spent by selecting from the following five options: “More than 5 days,” “3-5 days,” “1- 
2 days,” “Less than 1 day,” and “Not applicable.” The participant activities included counterparts of the seven 
activities and resources that were used to classify what the project offered (see, for example. Exhibit 5.13). In 
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■ For each of eight project-related activities addressed in the survey, participants 
who reported spending at least three days in the activity rated the project higher 
on both relevance and usefulness than did participants who spent less time in that 
activity (exhibit 5.14).^^ For example, those who participated for at least three 
days in conferences associated with a project gave an average rating of 4.15 for 
relevance and 3.87 for usefulness. In contrast, participants who spent less than 
three days in conferences gave average ratings of 3.82 and 3.63 for relevance and 
usefulness, respectively. All of the differences were statistically significant at the 
p<.05 level using a one-way ANOVA. 

■ Participating for at least three days in a larger number of activities was 
associated with higher relevance and usefulness ratings. There was a 
statistically significant relationship between the number of activities in which the 
participant spent three or more days and the rating of project relevance. The 
correlation between the number of different project activities in which the 
respondent participated for three or more days and the rating of relevance was 
+0.41, while the correlation with the rating of usefulness was +0.34. These 
correlation coefficients were statistically significant at p<.05. 



addition, participants were asked about time spent using “tools and other resources” provided by the project. This 
was a suitable item for differentiating among participants, since there was variation in the amount of time individual 
participants spent using tangible items that the project had provided. However, a corresponding item could not be 
used to differentiate among projects because the coding process revealed that all projects offered one or more 
tangible tools or resources to participants. 

Participant activities referred to in the text and presented in exhibit 5.14 are based on responses to the survey 
question on how much time individual participants had spent engaged in their own activities related to the project. 
These activity categories are not precisely aligned with the different set of codes used to categorize activities and 
resources that the Comprehensive Center offered, as reported on the project cover sheets. 



83 





Exhibit 5.14. Mean reievance and usefuiness ratings, by time respondents spent 
in each activity 



Type of activity or resource 


Time spent 


Relevance 


Usefulness 


Ongoing consuitation on 


3+ days (n=1,156) 


4.35 


4.06 


this topic 


<3 days {n=1, 11 5) 


3.81 


3.59 




Difference 


0.54* 


0.47* 




Significance 


p<.01 


p<.01 


Reviewing generai or 


3+ days (n=1,105) 


4.32 


4.07 


background materiais 
provided by the 
Comprehensive Center 


<3 days {n=1,534) 


3.78 


3.53 


Difference 

Significance 


0.54* 

p<.01 


0.54* 

p<.01 


Advance pianning 


3+ days (n=731) 


4.37 


4.08 




<3 days (n=1,311) 


3.91 


3.65 




Difference 


0.46* 


0.40* 




Significance 


p<.01 


p<.01 


Training 


3+ days (n=1,136) 


4.13 


3.93 




<3 days {n=1, 219) 


3.94 


3.72 




Difference 


0.19* 


0.21* 




Significance 


p<.01 


p<.01 


Task force meetings 


3+ days {n=581) 


4.29 


3.94 




<3 days (n=1,061) 


3.86 


3.61 




Difference 


0.43* 


0.33* 




Significance 


p<.01 


p<.01 


Conferences 


3+ days (n=1,219) 


4.15 


3.87 




<3 days (n=1,556) 


3.82 


3.63 




Difference 


0.33* 


0.24* 




Significance 


p<.01 


p<.01 


Foiiow-up and action pians 


3+ days (n=1,138) 


4.38 


4.04 




<3 days (n=1,307) 


3.81 


3.57 




Difference 


0.57* 


0.47* 




Significance 


p<.01 


p<.01 


Using toois and other 


3+ days (n=1,450) 


4.34 


4.10 


resources provided by the 
Center 


<3 days (n=1,305) 


3.68 


3.42 


Difference 


0.66* 


0.68* 




Significance 


p<.01 


p<.01 



NOTE: The types of activities and resources shown in this table are based on the types of activities presented 
in the participant survey and do not match categories presented in chapter 3 exhibits and exhibit 5.8, which 
were coded categories from the project cover sheets. 

* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: The relevance ratings among project participants who indicated that they spent 3 or more 
days receiving ongoing consultation or support was 4.35 and statistically significantly higher than the ratings 
from project participants who spent less time receiving ongoing consultation or support (3.81). 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all sample- 
eligible participants for the project. 
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Ratings by Participant Roies 

Were the sampled projects rated differently by participants who occupied different roles 
in their day-to-day jobs? The team investigated the possibility of a relationship between the 
participant’s place of employment or job responsibilities and the ratings he or she provided for a 
project’s relevance or usefulness. 

Given the intended emphasis of the Center technical assistance on building capacity at 
the state level, it is important to understand the extent to which the views of participants from 
state agencies might have differed from those of other participants. Of the sampled participants 
in RCC projects, 41 percent were employed by SEAs, 20 percent by schools, 15 percent by local 
education agencies (LEAs), 14 percent by intermediate education agencies, and 10 percent by 
other agencies or institutions.^^ Among the participants in the entire sample of CC projects, 59 
percent were employed by RCCs, 38 percent by SEAs, and 3 percent by schools or EEAs 
(exhibit 5.15). 



Exhibit 5.15. Where participants in RCC and CC projects were empioyed 
in 2006-07 



Employed by: 


Percent of participants 
RCC projects CC projects 


Total 


100 


100 


RCCs 


N/A 


59 


State education agency (SEA) 


41 


38 


Intermediate education agency 


14 


* 


Local education agency (LEA) 


15 


1 


School 


20 


2 


Other 


10 


* 



* Less than 0.5 percent. 

EXHIBIT READS: Among the surveys completed by participants in RCC projects, 41 percent 
were completed by respondents who were employed in a state education agency. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 
represented all sample-eligible participants for the project. 

■ Participants employed in SEAs did not give significantly different ratings for 
relevance or usefulness of either RCC or CC projects, compared with other 
participants (exhibits 5.16 and 5.17). The RCCs and CCs targeted assistance to 



Participants who were employed by agencies other than the SEA were considered “state-level” participants 
because they carried out state-level responsibilities such as membership in a school support team or state-level task 
force (see chapter 2 for more discussion of the definition of “state-level” participants). For this analysis, however, 
we differentiate among participants by their place of employment. 
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different clients corresponding with their respective roles: the RCCs provided 
technical assistance to the states and territories in their regions, whereas the CCs 
targeted RCCs for capacity building as well as working with RCCs in providing 
technical assistance to states. Ratings of relevance and usefulness did not differ 
between state-level staff and other client groups. For example, state-level and 
RCC staff did not rate the relevance and usefulness of CC projects differently. 



Exhibit 5.16. Mean reievance and usefuiness ratings for RCC projects, by agency 
where respondent worked 



Agency 




Relevance 


Usefulness 


State education agency 
(SEA) 


Yes {n=Q17) 


3.99 


3.68 


No (n=1,409) 


3.94 


3.68 




Difference 


0.05 


0.00 




Significance 


p=.13 


p=.93 


Intermediate education 


Yes (n^345) 


4.03 


3.67 


agency 


No (n=2,045) 


3.95 


3.68 




Difference 


0.08 


-.01 




Significance 


p=.07 


p=.90 


Local education 


Yes (n=365) 


3.87 


3.72 


agency (LEA) 


No (n^2,022) 


3.97 


3.67 




Difference 


-0.10* 


0.05 




Significance 


p=.03 


p=.30 


School 


Yes (n^469) 


3.88 


3.65 




No (n=1,918) 


3.98 


3.69 




Difference 


-0.10* 


-0.04 




Significance 


p=.02 


p=.44 


Other 


Yes (n=188) 


4.01 


3.67 




No (n^2,201) 


3.95 


3.68 




Difference 


0.06 


-0.01 




Significance 


p=.25 


p=.90 



* Difference statistically significant at p<.05 using a one-way ANOVA. 

NOTE: The category other included employees of institutions of higher education, nonprofits working with the 
state, and other RCCs. No CC staff members participated in RCC projects as recipients of technical assistance. 
EXHIBIT READS: Across RCC project participants, the mean relevance ratings provided by respondents who 
worked at state education agencies was 3.99. This was not statistically significantly different from the mean 
relevance rating of 3.94 provided by respondents who worked in other types of agencies. 

SCURCE: Surveys of project participants. Numbers reflect surveys returned, not individuals. Responses 
weighted so that respondents represented all sample-eligible participants for the project. 



86 




Exhibit 5.17. Reievance and usefuiness ratings for CC projects, 
RCC staff vs. state-levei staff 



Participant role 


Relevance 


Usefulness 


RCC staff (n^477) 


3.91 


3.73 


State-level staff 

(n=669) 


3.92 


3.81 


Difference 


-0.02 


-0.08 


Significance 


p=.74 


p=.14 



EXHIBIT READS: Among CC project participants, the mean relevance ratings provided by 
respondents who worked in RCCs was 3.91 . This was not statistically significantly 
different from the mean relevance rating of 3.92 provided by state-level respondents. 

SOURCE: Surveys of project participants. Numbers reflect surveys returned, not 
individuals. Responses weighted so that respondents represented all sample-eligible 
participants for the project. 



■ Participants who were employed in LEAs or schools (although they participated in 
Center assistance because they were working with the state) gave lower ratings for 
relevance to RCC projects, compared with other participants. In each case, the 
rating given was 0.10 points lower for relevance (exhibit 5.16) than was the rating 
given by other participants, and the difference was statistically significant at the 
p<.05 level using a one-way ANOVA. There were no other differences in ratings 
given to RCC projects by participants from different agencies. 

■ Comparing participants in CC projects who held different roles within RCCs, 
content specialists gave higher ratings for relevance and usefulness than did 
other RCC staff, while directors and assistant directors gave lower ratings 
(exhibit 5.18). The staff members who specialized in a content or topic area rated 
the CC projects 4.12 for relevance and 3.97 for usefulness, on average, compared 
with average ratings of 3.80 and 3.72 from other RCC staff. One possible 
explanation for this difference is that the CC emphasis on content areas was a 
good fit for the content-focused responsibilities of such staff members. The 
directors and assistant directors gave ratings of 3.54 for relevance and 3.52 for 
usefulness, compared with 4.04 and 3.90 ratings from other RCC staff All of the 
differences were statistically significant at the p<.05 level using one-way 
ANOVA. 
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Exhibit 5.18. Mean reievance and usefuiness ratings for CC projects, 
by RCC staff roie of respondent 



Participant role 




Relevance 


Usefulness 


State liaison 


Yes (n=315) 


3.91 


3.78 




No (n=346) 


3.93 


3.84 




Difference 


- 0.02 


- 0.06 




Significance 


p =.80 


p =.32 


Content or topic area 


Yes (n=254) 


4.12 


3.97 


specialist 




No (n=415) 


3.80 


3.72 




Difference 


0 . 32 * 


0 . 26 * 




Significance 


p <.01 


p <.01 


Center director or 


Yes (n=152) 


3.54 


3.52 


assistant director 




No (n=517) 


4.04 


3.90 




Difference 


- 0 . 50 * 


- 0 . 38 * 




Significance 


p <.01 


p <.01 


Other RCC staff 


Yes (n=187) 


3.75 


3.58 




No (n^482) 


3.99 


3.90 




Difference 


- 0.24 


- 0 . 32 * 




Significance 


p =.07 


p <.01 



* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: Among participants in CC projects who were employed by RCCs, the mean relevance 
rating provided by respondents who identified themselves as state liaisons was 3.91. 

SOURCE: Surveys of project participants. Numbers reflect surveys returned, not individuals. Responses 
weighted so that respondents represented all sample-eligible participants for the project. 



■ Participants who, in their regular jobs, spent more than 25 percent of their 
hours on NCLB-related state responsibilities gave higher ratings to Center 
projects than did those whose jobs focused less on NCLB (exhibit 5.19). Because 
the Centers’ charge focused on assistance related to NCLB, the team hypothesized 
that participants whose regular job responsibilities focused most heavily on that 
law might give relatively higher ratings for relevance and usefulness of the 
sampled projects. We found that those who spent at least one quarter of their 
work time on NCLB gave significantly higher relevance and usefulness ratings to 
the projects, compared with those who spent 25 percent of their time or less 
(p<.01 for differences in both relevance and usefulness using ANOVA). 



Participants were asked in the survey to indicate what percentage of their hours on the job they spent working on 
any of the specific state-level responsibilities related to NCLB. They were given a choice of four percentage ranges: 
0-25 percent, 26-50 percent, 51-75 percent, and 76-100 percent. 
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Exhibit 5.19. Mean reievance and usefuiness ratings, by percent of time 
respondent spent on NCLB-reiated tasks in his/her job 



Percent of time spent on 
NCLB-related tasks 


Relevance 


Usefulness 


More than 75 percent (n=908) 


4.05 


3.87 


51 to 75 percent (n=541) 


3.98 


3.78 


26 to 50 percent (n-493) 


3.94 


3.72 


25 percent or less (n=476) 


3.72 


3.47 


Range 


0 . 33 * 


0 . 40 * 


Significance 


p<.01 


p<.01 



* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: The mean relevance rating of 4.05 provided by project participants whose job 
responsibilities included NCLB-related tasks more than 75 percent of the time was statistically 
significantly than the mean relevance ratings provided by project participants whose NCLB-related 
job responsibilities took up less of their time. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all 
sample-eligible participants for the project. 
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Appendix A 

Comprehensive Center Lead Grantee 
and Subgrantee Organizations 




Comprehensive Center Lead Grantee 
and Subgrantee Organizations 



Center name 


Lead grantee and subgrantees 


Alaska Comprehensive 
Center 


■ 


Southeast Regional Resource Center (SERRC of Alaska) 
o Northwest Regional Educational Laboratory (NWREL) 


Appalachia Regional 
Comprehensive Center 


■ 


Edvantia 

o George Washington University - Center for Equity and Excellence 
in Education (CEEE) 

o Eastern Stream Center on Resources and Training (ESCORT) 
o National Center for Family Literacy (NCFL) 
o University of North Carolina at Greensboro - SERVE Center 
o Southern Regional Education Board (SREB) 


California 

Comprehensive Center 


■ 


WestEd 

o American Institutes for Research (AIR) 
o School Services of California 


Florida and Islands 
Comprehensive Center 


■ 


Educational Testing Service (ETS) 
o University of South Florida - David C. Anchin Center 
o Edvantia 

o Eastern Stream Center on Resources and Training (ESCORT) 
o Florida Association of School Administrators (FASA) 
o International Center for Leadership in Education (ICLE) 
o JLM Professional Education Services 
o Robin Wheeler, LLC 


Great Lakes East 
Comprehensive Center 


■ 


Learning Point Associates 
o RMC Research Corporation 
o American Institutes for Research (AIR) 
o Center for Applied Linguistics (CAL) 
o University of Michigan - Consortium for Policy Research in 
Education (CPRE) 


Great Lakes West 
Comprehensive Center 


■ 


Learning Point Associates 
o American Institutes for Research (AIR) 
o Mid-continent Research for Education and Learning (McREL) 
o University of Wisconsin - Wisconsin Center for Educational 
Research (WCER) 

o University of Michigan - Consortium for Policy Research in 
Education (CPRE) 


Mid-Atlantic 
Comprehensive Center 


■ 


George Washington University - Center for Equity and Excellence in 
Education (CEEE) 
o Edvantia 

o Eastern Stream Center on Resources and Training (ESCORT) 
o Group Jazz 

o Southern Regional Education Board (SREB) 


Mid-Continent 
Comprehensive Center 


■ 


University of Oklahoma 
o Northrop Grumman Information Tech 
o Acclon Social Comunitaria 

o Mountain Plains Regional Resource Center (MPRRC) 
o North Central Regional Resource Center (NCRRC) 
o Southeast Regional Resource Center (SERRC of Alabama) 


New England 
Comprehensive Center 


■ 


RMC Research Corporation 
o Education Development Center (EDC) 
o WestEd 

o Education Alliance at Brown University 
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Comprehensive Center Lead Grantee 
and Subgrantee Organizations (continued) 



Center name Lead grantee and subgrantees 



New York 

Comprehensive Center 


■ 


RMC Research Corporation 
o Education Development Center (EDC) 
o WestEd 

o Education Alliance at Brown University 
o United Federation of Teachers Teacher Center (UFTTC) 


North Central 
Comprehensive Center 


■ 


Mid-continent Research for Education and Learning (McREL) 


Northwest Regional 
Comprehensive Center 


■ 


Northwest Regional Educational Laboratory (NWREL) 
o RMC Research Corporation 


Pacific Comprehensive 
Center 


■ 


Pacific Resources for Education and Learning (PREL) 


Southeast 

Comprehensive Center 


■ 


Southwest Educational Development Laboratory (SEDL) 
o Center for the Education and Study of Diverse Populations 
(CESDP) 


Southwest 

Comprehensive Center 


■ 


WestED 

o RMC Research Corporation 
o American Institutes for Research (AIR) 


Texas Comprehensive 
Center 


■ 


Southwest Educational Development Laboratory (SEDL) 
o Center for the Education and Study of Diverse Populations 
(CESDP) 


Assessment and 
Accountability Content 
Center 


■ 


WestEd 

o National Center for Research on Evaluation Standards, and Student 
Testing (CRESST) 


Center for Innovation 
and Improvement 


■ 


Academic Development Institute (ADI) 

o Temple University - Center for Research in Human Development 
and Education (CRHDE) 
o Little Planet Learning 


Center on Instruction 


■ 


RMC Research Corporation 
o Florida Center for Reading Research (FCRR) 
o RG Research Group 
o Horizon Research 

o Texas Institute for Measurement, Evaluation and Statistics (TIMES) 
o University of Texas at Austin - Vaughn Gross Center for Reading 
and Language Arts 


National Comprehensive 
Center on Teacher 
Quality 


■ 


Learning Point Associates 
o Education Commission of the States (ECS) 
o Educational Testing Service (ETS) 
o Vanderbilt University 


National High School 
Center 


■ 


American Institutes for Research (AIR) 
o Learning Point Associates 
o MDRC 

o National Center for Educational Accountability (NCEA) 
o WestEd 



EXHIBIT READS: The Alaska Comprehensive Center was operated by the lead grantee organization named the 
Southeast Regional Resource Center (of Alaska) with support from the subgrantee organization named the 
Northwest Regional Educational Laboratory. 



Source: U.S. Department of Education; individual RCC and CC websites. 

A-2 







Appendix B 
Study Sample 




study Sample 



This appendix section presents the Project Inventory Forms that were collected to 
identify the sample of projects to be included in the study as well as the documents sent 
to Centers requesting materials for expert panel review of sampled projects. 



Project Inventory Form 

The Project Inventory Forms (PIF) was developed by the study team as a standard 
way for Centers to provide an inventory of their work for each program year. When asked 
to complete the inventory forms, Comprehensive Center Directors were informed about 
the specific project year for which information was being gathered (July 1, 2006 - June 
30, 2007) as well as the purpose of this information to be used in the selection of a 
sample of projects for review of quality by expert panels to be conducted for the national 
evaluation. When completing the PIF, each Center was asked to nominate several 
projects that they believed best represented the work undertaken by that Center. 

Additional projects were purposefully sampled from the remaining projects on each 
center’s inventory to reflect each center’s overall portfolio of work, as well as the work of 
all regional or content centers in key topic areas. A sample PIF was provided including 
examples of the kinds of projects that should be listed by the centers and examples of 
activities and resources defining each project level of effort - major, moderate, minor. 
Examples of some projects, activities, and deliverables that should not be included on the 
inventory form were also provided, including the following; 

■ Training or professional development for Comprehensive Center staff 

■ Work on coordinating committees within the Comprehensive Center 
network 

■ Annual needs assessment activity or negotiations with states, unrelated to 
specific projects 

■ Other internal working meetings or documents 

When completing the PIF, each center provided a list of each project under the 
appropriate topic heading (see list of topic areas and definitions included in PIFs, on p. B- 
11). For each listed project. Centers followed instructions to provide information 
describing the projects and listing the activities and deliverables associated with each 
project (see section of PIF, Instructions for Completing an Inventory of Projects, p. B-4 - 
B-8). 



Letters Requesting Materials for Expert Panel Review from Centers 

Each Center received a notification letter from the evaluation team that listed the 
projects from their PIE that had been selected for expert panel review followed by 
guidance in assembling the materials for panel review, compiling lists of project 
participants for the participant survey, and completion of the cover sheet for each 



B-1 




sampled project. The initial request for materials letter and supporting information starts 
on page B-19. The evaluation team reviewed the materials submitted by the Centers for 
accuracy and completeness and in cases where there were missing or incomplete 
materials or where further explanation was needed, followed up with Centers using the 
follow-up memo provided later in this appendix starting on page B-26. 
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Project Inventory Form 



Comprehensive Center Project Inventory Form 



Dear Center Director, 

Enclosed are a Project Inventory Form and instructions to be used to compile a complete 
list of your Center’s projects during the current project year (July 1, 2006-June 30, 2007). 

The national evaluation will use the completed project inventories to select a sample of 
projects for review by expert panels. Please designate several projects that you believe best 
represent the work of your Center; these projects will be included among the sample of projects 
to be reviewed by expert panels. These panels will rate nominated and sampled projects for their 
quality. 

Please complete a draft inventory to review with a member of the national evaluation 
team during the site visit for the evaluation, scheduled to take place between April and June 
2006 when [EVALUATION STAFF MEMBER] visits your Center. A final version of the 
project inventory should be completed as soon as possible after the site visit, but no later than 
June 30, 2007. Please return the completed inventory form to your evaluation liaison, [name] at 
@ • 

If you have any questions about the inventory or the instructions for completing it, please 
contact at 1 -xxx-xxxx or by email at @ . 

Thank you for your continued support of the national evaluation of the comprehensive 
center program. 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid OMB control number. The valid OMB control number for this data collection 
instrument is 1850-8023. The time required to complete these worksheets is estimated to average 16 hours per 
response, including the time to review instructions, search existing data sources, gather the data needed, and fdl in 
the form. Send comments regarding this burden estimate or any other aspect of this collection of information, 
including suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 



B-3 







Project Inventory Form 



Instructions for Completing an Inventory of Projects 

Each Comprehensive Center will prepare an inventory of all of the projects that were 
active during the current grant period. The first inventory will cover the period from July 1 , 2006 
through June 30, 2007. The second inventory will cover materials from July 1, 2007 through 
June 30, 2008, and the third inventory will cover July 1, 2008 through June 30, 2009. 

Projects that began in the current project year but will not be complete by June 30 should 
be entered in the inventory and noted as “ongoing.” 

Purpose 

This inventory will serve as the sampling frame for the expert panel reviews to be 
conducted for the national evaluation. A sample of projects will be selected in two ways. Each 
Center will nominate several projects that they believe best represents the work undertaken by 
that Center. The remaining projects will be purposefully sampled from the remaining projects on 
each Center’s inventory. Samples will be drawn to reflect each Center’s overall portfolio of 
work, as well as the work of all Regional or Content Centers in key topic areas. Items selected 
for review will be rated by an expert panel for their quality. 

If a project is nominated or sampled for expert panel review and rating, the evaluation 
team will ask your Center to collect and transmit all of the documents and other artifacts 
associated with that project (meeting agendas, briefing books, meeting summaries, training 
materials, white papers, web resources, etc.) for the expert panel to review. 

Identifying Projects for Inclusion in the Inventory 

The unit of analysis for the expert review panels will be the project, Eor the purposes of 
this inventory, a “project” will ordinarily comprise a group of closely related activities and/or 
deliverables designed to achieve a specific outcome for a specific audience. The Content 
Centers have some projects that consist of the development of a single product or deliverable, 
rather than a group of deliverables, but a project that comprises only a single product or 
deliverable will be the exception rather than the rule. The inventory should include all of the 
projects developed or delivered by the Center, including those developed or delivered in 
collaboration with other Centers, during the reporting period. 

Because each project listed in the inventory could potentially be sampled for expert panel 
review, each project shown as an entry (or row) in the inventory form should be a relatively 
complete project that can be understood and rated on its own by expert panel members who may 
not know anything about other aspects of the Center’s work. Although a single project may 
include a number of deliverables and activities, it will be designed to achieve a specific outcome 
and address (in almost all cases) a single topic. Where a group of activities and deliverables can 
be divided up into separate projects, each constituting a complete and coherent whole, the Center 
should list these as separate projects in the inventory. 

The following criteria should guide the Centers as they identify projects (and their 
associated activities and deliverables) for recording in the inventory form. They have been 
developed to ensure that projects constitute units that are large enough for review and rating, but 
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Project Inventory Form 



focused enough for coherence. Each project entered in the inventory should satisfy all three of 
the following criteria: 



■ Complete and coherent whole. Each project listed in the inventory should be 
able to stand on its own in an expert panel review. Avoid listing activities and 
deliverables as separate projects in the inventory if they cannot be understood or 
evaluated without information about related activities and deliverables. Eor 
example, a training event may require extensive planning (e.g., needs assessment, 
materials development) and follow-up activities (e.g., evaluation of the training, 
consultation on action plans). These planning and follow-up activities would be 
very difficult for a panel to rate in the absence of information about the event 
itself. Therefore the Center should list these planning and follow-up activities and 
the event itself as a single project (one row) on the inventory form. Each phase of 
the project — planning, the event itself, and follow-up — ^will be described briefly 
in the “activities and deliverables” column. Similarly, ongoing work with a state- 
level task force should be listed as a single project rather than each meeting of the 
task force being listed as a project. 

■ Common intended outcome. Where a cluster of activities and deliverables is 
designed by the Center to lead to the same outcome for the same audience(s), 
those activities and deliverables should be grouped as one project in the inventory 
form. On the other hand, where a set of activities is intended to produce more than 
one distinct outcome — for example, helping the state to develop a strategic plan 
for improving reading instruction, and helping the same state recruit and train 
literacy coaches — those two activities should be listed as separate projects. Where 
the Center replicates the same set of activities in each of several different states, 
that set of activities should be listed as a single project if the intended outcomes 
and processes do not differ materially from one state to another. Where the 
intended outcomes do differ substantially from state to state, the work in each 
state should be listed as a separate project. 

■ Topic area focus. With few exceptions, a project addresses just one of the 14 topic 
areas described in Exhibit A at the end of this document (e.g., state systems of 
support, reading/language arts). Where it is possible to divide a group of related 
activities into two different projects according to the topic area addressed. Centers 
should do so. Eor example, a regional forum on interventions for low-performing 
students in reading and mathematics that offers separate strands of sessions in each 
subject area should be listed as two different projects, one under reading/language 
arts and one under mathematics. In this case, all of the sessions on reading 
interventions would stand on their own as a complete and coherent whole for rating 
by an expert panel and should be listed as a separate project; the same would be true 
for the sessions on mathematics. (If a project cuts across topic areas and the 
activities and deliverables which it comprises cannot be divided up by topic area 
into complete and coherent units that would make sense to a review panel, the 
project should be listed under the most relevant topic area with a note cross- 
referencing other topics, following the instructions for column IV below.) 
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Project Inventory Form 



The sample inventory form at the end of this packet includes examples of the kinds of 
projects that should be listed by the Centers. 



Some projects, activities, and deliverables should not be included on the inventory form 
at all. These include: 



■ Training or professional development for Comprehensive Center staff 

■ Work on coordinating committees within the Comprehensive Center network 

■ Annual needs assessment activity or negotiations with states, unrelated to specific 
projects 

■ Other internal working meetings or documents 



Completing the Inventory Form 

Centers should use the attached form to complete their inventory of projects. Sample 
projects and examples of entries can be found in Exhibit B. 

Centers might find it useful to review their annual project plans, technical assistance 
plans, management plans or technical assistance logs as a starting point since those documents 
typically provide an overview of the various projects and activities that were planned for or 
conducted during the year. 

Once the inventory form is complete. Centers should designate which projects they would 
like to nominate for inclusion in the sample of projects reviewed by the expert panels, by inserting 
** after the name of the project in Column II. Centers should designate several projects. 

List each project under the appropriate topic heading. A list of topic headings, with 
definitions, is attached at the end of this document in Exhibit A. 

Directions for completing each column are as follows: 
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Project Inventory Form 



Topic Area Headings 

I, Project number 

II, Name 

III, Description 

IV, Additional topics 
addressed? 

V, Activities and 
Deliverables 



VI, Start Date 

VII, End Date 



List items under the appropriate topic area heading. Where a project 
fits under more than one topic heading, list it once, under the topic 
heading that is most relevant. Note the project’s relevance to other 
topics in Column IV. Add rows to the table as needed. Leave rows 
blank if your Center does no work in a particular topic area. 

Assign consecutive numbers to each item listed in the inventory. 
(Centers may want to complete this column as a final step, after all of 
the items have been entered in the inventory.) 

Enter the name of the project. Projects nominated by Center staff for 
review should be followed by ** in this column. 

Provide a concise description of the project. (See examples for 
appropriate level of detail.) 

If the project addresses more than one topic area, note that here. Use 
the list of topic areas provided in the appendix (this list corresponds 
with the row headings in the inventory). Entering the appropriate 
number from the list will save space. 

Eist all of the activities and deliverables associated with the project. 
(Eor products, include exact title, if applicable. Eor services, include 
location and type of participant.) Specific activities and deliverables 
may include: 

Meetings/conferences (includes items such as workshops, 
conferences, institutes, forums, webinars) 

Expert consultation/technical assistance (includes items such as 
assistance completing reports or applications, review of state plans, 
needs assessments, audits) 

Facilitation/support of working groups or teams (includes items 
such as planning meetings, participation in meetings, drafting 
summary documents) 

Guidance/information resources (includes items such as policy or 
issue briefs, fact sheets, congressional testimony, resource guides, 
planning tools, field guides, benchmarking rubrics, handbooks, 
exemplars, literature reviews/summaries, annotated bibliographies, 
case studies, websites) 

Training (includes items such as professional development 
materials/services, software, training materials) 

Other, specify 

Enter the start date for the project, including month and year. 



Enter the end date for the project, including month and year. 
If the project is currently ongoing, enter the note “ongoing.” 
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Project Inventory Form 



VIII: Major, 
Moderate or Minor 
Project 

IX: Target state(s), 
region(s), or regional 
center (s) 



X: Collaborations and 
Sources: Other CCs 



XI: Collaborations 
and Sources: Other 
TA Providers 



Indicate whether you consider this a major, moderate or minor project 
in terms of the relative level of effort and/or resources devoted to it. 



Regional Comprehensive Centers serving multiple states should note 
which state(s) participated in each project. Regional Centers serving a 
single state (Alaska, California, New York, and Texas) should note 
which region(s) within the state participated (e.g.. New York City vs. 
rest of state). Content Centers should specify which Regional Centers 
participated in the project. 

If the Center used materials developed by one of the Content Centers 
in the course of designing or delivering its own services, list this 
Content Center as a source. 

If the Center collaborated with another Regional Center or Content 
Center on the design, development and/or delivery of products and 
services, list this Center as a collaborator. 

Note collaborations with other Comprehensive Centers only in this 
column. 

Note collaborations with other technical assistance providers, such as 
Regional Educational Laboratories or universities, in this column. 

If the Center used materials developed by another TA provider in the 
course of designing or delivering its own services, list this provider as 

a source. 

If the Center collaborated with another TA provider on the design, 
development and/or delivery of products and services, list this 
provider as a collaborator. 
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National Evaluation of the Comprehensive Technical Assistance Centers 

Inventory of Projects 



Center Name: 



I. No. 


II. Name 


III. Description 


IV. Additional 
topics addressed? 

(From list in 
Exhibit A — enter 
appropriate 
number) 


V. Activities and 
Deliverahles 


VI. Start Date 


VII. End 
Date (enter 
“ongoing” 
if project is 
not 

complete) 


VIII. 
Major, 
moderate, 
or minor 
project 


IX. Target 

state(s), 

region(s) 

within a 

state, or 

Regional 

Centers(s) 


X. 

Collab- 

orations 

and 

Sources: 

Other 

CCs 


XI. 

Collah- 

orations 

and 

Sources: 
Other TA 
Providers 


1. State Systems of Support for Schools Identified for Improvement 














































2. Building District/Local Capacity to Su| 


pport School Improvement, or Address Corrective Action and Restructuring 














































3. Assessment (excluding assessment of special education students or ELL students) 
















































4. Reading/Language Arts Curriculum, Instruction, and/or Professional Development (includes Adolescent Literacy) 














































5. Mathematics Curriculum, Instruction, and/or Professional Development 














































6. Other Content Area Curriculum, Instruction, Professional Development, Standards-Based and Research-Based Instructional 


1 Frameworks 














































7. High School Reform 














































8. Special Education 














































9. English Language Learners 
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I. No. 


II. Name 


III. Description 


IV. Additional 
topics addressed? 

(From list in 
Exhibit A — enter 
appropriate 
number) 


V. Activities and 
Deliverables 


VI. Start Date 


VII. End 
Date (enter 
“ongoing” if 
projeet is 
not 

eomplete) 


VIII. 
Major, 
moderate, 
or minor 
project 


IX. Target 

state(s), 

region(s) 

within a 

state, or 

Regional 

Centers(s) 


X. 

Collab- 

orations 

and 

Sources: 

Other 

CCs 


XI. 

Collab- 

orations 

and 

Sources: 
Other TA 
Providers 


10. Highly Qualified Teachers 














































11. Parent and Community Involvement 














































12. Supplemental Educational Services 














































13. Other NCLB-Related TA 














































14. Other 















































**Projects nominated by the Center for review by expert panels 





Project Inventory Form 



Exhibit A 

Topic Areas and Definitions 



Topic Area 


Definition 


1 . State Systems of 
Support for Schools 
and Districts 
Identified for 
Improvement 


Work that supports an SEA and/or its service delivery system carry out 
state-level responsibilities related to supporting district and schools 
identified for improvement. Service may target School Support Teams 
(SSTs), Distinguished Educators (DEs), or regional education service 
centers (RSCs, ESCs). 


2. Building 
District/Local 
Capacity to Support 
School 

Improvement, or 
Address Corrective 
Action and 
Restructuring 


Includes any work designed to build district- and school-level capacity 
to carry out school improvement and to make AYP, as required under 
NCEB, including remedies taken for schools or districts in “corrective 
action” or “restructuring.” 


3. Assessment 


Design and implementation of local assessment systems, such as 
benchmark assessments aligned to state assessments; classroom 
assessments designed to support instructional change; or support for 
analysis and use of data to drive instructional change. Includes 
consultation on the design and implementation of state assessment 
systems. Does NOT include alternate assessments for SPED or EEE. 


4. Language Arts 
Curriculum, 
Instruction, and 
Professional 
Development 


Includes all work at all levels that addresses ELA, reading, and literacy. 
Includes projects on adolescent literacy. May include work related to 
curriculum alignment, research-based models or programs, standards 
and policy reviews, and teacher professional development. 


5. Mathematics 
Curriculum, 
Instruction, and 
Professional 
Development 


Includes all work at all levels that addresses mathematics and 
numeracy. May include work related to curriculum alignment, 
research-based models or programs, standards and policy reviews, and 
teacher professional development. 


6. Other Content Area 
Curriculum, 
Instruction, and 
Professional 
Development; 
Standards-Based and 
Research-Based 
Instructional 
Frameworks 


Includes all work at all levels that addresses specific content areas other 
than language arts or math, such as science and social studies. May 
include work related to curriculum alignment, research-based models or 
programs, standards and policy reviews, and teacher professional 
development. Includes work on curriculum alignment and policies 
related to academic standards that are NOT content area-specific. 
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Project Inventory Form 



Topic Area 


Definition 


7. High School Reform 


Includes work related to smaller learning communities, ninth grade or 
high school transition, dropout prevention, high school policies. 


8. Special Education 


Includes all work related to special education, such as alternate 
assessments for the “1 percent and 2 percent students” or 
instructional/assessment strategies that target needs of students with 
lEPs. 


9. English Eanguage 
Eeamers 


Includes all work related to EEE (i.e. EEP) students, such as work on 
Annual Measurable Achievement Objectives (AMAOs) for EEE 
students and instruction/assessment of EEE students. 


10. Highly Qualified 
Teachers 


All work addressing the NCEB requirement that states have 100 
percent of teachers licensed to teach in their subject area and 100 
percent of paraprofessionals with 2-years of college. 


11. 




12. Supplemental 

Educational Services 


Includes work related to helping SEAs evaluate and monitor 
Supplemental Education Services, as well as setting up systems for 
notification and outreach to districts, schools, and parents. 


1 3 . Other NCEB-related 
TA 


Completing miscellaneous federal applications or reports, convening 
SEA staff to address miscellaneous NCEB topics; ongoing consultation 
on NCEB topics, including migrant education. 


14. Other 


Includes projects that do not fit under any other topic area, including 
websites, membership on SEA committees, and ongoing phone support. 
Includes projects building general skills and knowledge of leaders at all 
levels. Includes projects that assist clients with the review, collection, 
and translation of research, where there is no specific topic. If the 
research addresses a specific topic area, put the project in the 
appropriate row. Does not include internal or network meetings — these 
types of activities should not be included in the project inventory. 
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Exhibit B 

Sampie Project Inventory 



Center Name; 

Note: All examples in the table were taken from Year 2 baseline management plans. 



I. No. 


II. Name 


III. Description 


IV. 

Additional 

topics 

addressed? 


V. Activities and Deliverables 


VI. 

Start 

Date 


VII. 

End 

Date 


VIII. 
Major, 
moderate 
or minor 
project 


IX. 

Target 

state(s), 

region(s), 

or 

centers(s) 


X. 

Collab- 

orations 

and 

Sonrces: 

Other 

CCs 


XI. 

Collab- 

orations 

and 

Sonrces: 
Other TA 
Providers 


1. State Systems of Snpport for Schools Identified for Improvement 


1. 


State TA 
system for PI 
distriets and 
sehools 


Designed an 
integrated SEA TA 
system that reaehes 
“program 
improvement” 
distriets and sehools 




• Convened coordinating council; 

• Created inventory of existing TA 
efforts; 

• Helped create TA plan; 

• Gave PD and TA to school support 
teams 


7/2006 


Ongoing 


Moderate 








2. 


Statewide 
system of 
support** 


Supported 
development of 
Regional Sehool 
Serviee Centers as a 
system of support for 
addressing NCLB 




• Helped create a plan to guide the 
Regional Network Strategy for the 
next five to seven years; 

• Developed the next RFP for the 
RSSCs; 

• Supported RSSC implementation; 

• Developed and helped deliver PD; 

• Developed a protocol for collecting 
information on implementation 


7/2006 


6/2007 


Major 


State A 






3. 


“Signifieant 
Change in 
Sehool 
Improvement 
and 

Restmeturing” 


Developed a modular 
handbook and 
workshop on 
implementation of 
fast-paeed signifieant 
sehool improvement, 
ineluding 
restmeturing 


2 (Local 
capacity) 


• Drafted 8 modules for handbook to 
be used with SEA systems of 
support; 

• Presented at institute; 

• Revised and added workshop 
materials; 

• Consulted at CC A regional 
meeting 






Major 


All 

Regional 

CCs 


Regional 

CCA 
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Support to 


Helped the SEA 


School 


School Improvement 


Improvement 


Team to identify 


Team 


resource and design 




training for schools in 




“monitored” and 




“priority” statuses 




7/2006 6/2007 Minor 



2. Building District/Local Capacity to Support 



5. 



Attended meetings organized by 
SIP team and provided refleetive 
syntheses; 

Brokered resourees and serviees 
available through RRC and CC 
networks 



School improvement, or Address Corrective Action and Restructuring 





District tools 
for monitoring 
schools 


Helped the SEA 
develop processes and 
tools for districts to 
use in monitoring the 
implementation of 
school restructuring 
plans 


Leadership 


Designed and 


Institute on 


implemented a PD 


Helping 


plan to support the 


Districts Assist 


Local Education 


Low- 


Agency Assistance 


Performing 

Schools 


Program 


Guidance to 


Assisted SEA in 


districts on 


developing guidance 


restructuring 


for districts with 


schools 


schools in 
restructuring 




• Prepared draft processes and tools 
for monitoring; 

• Met with SEA school improvement 
staff to review/revise tools 


7/2006 


10/2006 


Minor 


• Convened planning team to design 
the Leadership Institute; 

• Helped conduct institute; 

• Provided ongoing support to 
participants as they work with 
districts and schools 


7/2006 


6/2007 


Moderate 


• Assisted state in writing guidance 
protocol; 

• Piloted guidance; 

• Collected and prepared research; 

• Provided initial training to districts 
with schools in restracturing 


7/2006 


12/2006 


Moderate 




3. Assessment (excluding assessment of special 



Growth Built SEA 

models understanding of 

assessment to help in 
deeision making about 
aeeountability growth 
models 



education students or ELL students) 




• Identified state needs and create 


7/2006 


Ongoing 


Major 


All 


plan for growth model; 








states in 


• Assisted states accepted as pilot 
states; 

• Assisted ongoing development for 








region 


other states 
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Assessment 

data 

analysis** 



Built SEA and other 
staff eapaeity by 
helping them train 
distriets and sehools 
in analyzing state 
assessment results and 
using them for 
instruetional planning 



4. Reading/Language Arts Curriculum, lustructiou, aud/or Professioual Developmeut (iucludes Adolesceut Literacy) 



• Met with SEA staff to plan PD; 


5/2006 


6/2007 


Major 


State A 


Content 


• Helped host Statewide Assessment 
Conference; 

• Continued to plan and offered 
training in assessment analysis; 

• Established a cadre of educators to 










CCA 


provide PD to districts 













Adolescent 

Literacy 

Cadres 



Effective 
Leadership in 
Literacy 



Organized and 
delivered training of 
adolescent literacy 
cadres made up of 
SEA staff 



Built SEA capacity to 
support “effective 
leadership in literacy 
for grades 6-12” 




• Convened SEA work teams to 
analyze research; 

• Reviewed literacy diagnostic tools; 

• Customized a literacy assessment; 

• Conducted PD for SEA work teams 
on customized assessments 


10/200 

6 


6/2007 


Moderate 


All 

states in 
region 


Content 
CC A; 
Content 
CC B; 
Content 
CCC 


• Convened work group to discuss 
research; 

• Provided SEA with PD on Strategic 
Instruction Model; 

• Developed state literacy plan 


7/2006 


6/2007 


Major 


State A; 
State B 


Content 

CCA 



5. Mathematics Curriculum, lustructiou, aud/or Professioual Developmeut 



K-2 

Mathematics 

Inventory 



Developed and 
provided PD to a 
cadre of educators to 
help schools 
implement the state 
K-2 Mathematics 
Inventory 






• Helped SEA design PD; 


7/2006 


6/2007 


Major 


State A 


Content 


• Helped design team pilot 
workshop; 

• Revised PD; 

• Helped deliver regional trainings 










CCA 


prioritized for schools in 
improvement; 












• Reviewed feedback to redesign PD 
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7. High School Reform 

13. Implementing Developed capacity of 
High School SEA to implement 

Task Force recommendations of 

Report HS Task Force Report 

related to high school 
redesign 



8. Special Education 

14. Assessing Supported SEAs in 10 (ELL) 

special needs establishing 

students assessment and 

accountability systems 
that include and 
support special 
education, ELL, and 
low-income students 

15. Developing an Assisted SEAs in 3 

Alternate developing Alternate (Assessmen 

Assessment Academic t) 

Assessments for the 
“1” of students who 
have significant 
cognitive disabilities 

16. Research Published Research 12 (HQT) 

Synthesis on Synthesis 1 focusing 

Special Needs on teaching special- 

Students needs students 



Work group and Steering 
Committee met to consider task 
force recommendations, research, 
best practice, and review of prior 
initiatives; 

Identified “knowledge 
managemenf ’ components needed 
to support state implementation; 
Identified policies and practices 
needed to promote redesign; 
Developed multi-year 
implementation plan. 



Collected information from Content 
CC A on research- based practices; 
Conducted SEA Needs 
Assessment; 

Provided regional training(s) on 
Scientifically Based Research on 
assessment and accountability 
systems for special needs students 



Identified needs and broker 
assistance in developing 
assessment; 

Provided TA on development of 
assessment; 

Provided ongoing TA on 
implementation 



Identified and evaluated existing 
research and syntheses; 

Conducted gap analysis to identify 
research synthesis needs; 

Wrote synthesis (50-60 pages); 
Posted on on-line resource forum. 





Major 



All 

Regional 

CCs 
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9. English Language learners 

17. District Helped develop a 13 (Parents) 

guidance on comprehensive 

the education framework for 

of ELL educating English 

students Language Learners to 

guide district work, 
including guidance on 
the use of formative 
assessments to 
improve instruction 
and on family and 
community 
engagement 

18. Addressing Assisted SEA in 

AMAOs aligning Annual 

Measurable 
Achievement 
Objectives, standards, 
and assessments for 
ELL students 

10. Highly Qualified Teachers 

19. “TQ Source Developed online 

Project” resource 

clearinghouse that 
provides clients with 
policy and research 
data specifically 
related to teacher 
quality, as addressed 
in NCLB 



• Facilitated meetings of the SEA 7/2006 6/2007 Major State A Content 

task force responsible for CC A 

developing and disseminating a 

comprehensive framework for the 
education of ELLs; 

• Assisted in integrating effective 
formative assessment practices for 
ELL students; 

• Assisted in formulating guidance 
on ELL family and community 
engagement strategies; 

• Provided consultation and 

resources from Content CC A 

• Conducted a study on alignment of 8/2006 12/2006 Minor State A; 

English Language Proficiency State B; 

standards for grades K-12 with the State C 

state assessment for ELL students 

• Helped state reconfigure AMAOs 
to align with state test 



• Developed “interactive data tool” 7/2006 Ongoing Major All 

based primarily on NCES data; Regional 

• Updated “policy database” for CCs 

state -by- state policy areas, 

including policies related to PD and 
teacher prep, recruitment, retention, 
certification, etc.; 

• Published next issue of “Tips and 
Tools: Emerging Strategies to 
Enhance Teacher Quality”; 

• Enhanced “Publications database” 
focusing on teacher quality 
research 
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20. 


Teacher 
Preparation 
and Licensure 
in literacy 


Assisted SEA in 
reviewing teacher 
preparation and 
licensure requirements 
related to literacy 


4 

(Language 

arts) 


• Assisted the Task Force on 
Licensure/Professional 
Development in revisiting 
preparation and licensure 
requirements related to literacy 
instruction for school leaders and 
teachers; 

• Discussed findings and possible 
next steps related to engagements 
with representatives from institutes 
of higher education 


10/200 

6 


2/2007 


Minor 


State A 






12. Supplemental Educational Services 


21. 


“State 

Evaluation of 

Supplemental 

Educational 

Services” 

Manual 


Updated “State 
Evaluation of SES” 
manual and assist with 
its use by Regional 
CCs 




• Drafted updates to SES evaluation 
manual 

• Presented at September 2006 
Institute 

• Revised based on evaluations and 
usage 

• Submitted evaluation report to ED 


9/2006 


4/2007 


Major 


All 

Regional 

CCs 


Regional 

CCA 




13. Other NCLB-related TA 


22. 


Enhancing 

Schoolwide 

Planning 

Materials 


Assisted SEA in 
updating resources 
and training for 
district Title I 
directors on the topics 
of schoolwide 
planning, plan 
implementation, and 
district monitoring of 
school plans 




• Flelped update T1 schoolwide 
application and evaluation rubric; 

• Ensured that North Central 
Association Commission on 
Accreditation and School 
Improvement school improvement 
processes include schoolwide plan 
components; 

• Reviewed need assessment results 
from disiricl/schools; 

• Updated materials based on needs 


7/2006 


6/2007 


Major 


State A 







**Projects nominated by the Center for review by expert panels 














Request for Materials for Expert Panel Reviews 




Decision Information Resources, Inc. 



REQUEST EOR MATERIAES EOR EXPERT PANEE REVIEWS 



September 10, 2007 



Dear [Center Direetor]; 

I am happy to report that we have completed the sampling process for your site and have 
selected the following (n) projects for review: 



Project Name 


Project Description 































These projects will be rated for technical quality by panels of expert reviewers, and for 
relevance and usefulness through surveys of project participants. 

Assembling Panel Review Materials 

Attached you will find a Request for Materials form that includes some basic guidance 
for assembling materials for the panel review process. This form also includes the project 
cover sheet, which should be completed for each project. This cover sheet will be 
provided to the review panelists and is an important source of background and contextual 
information about the project. 

To facilitate the review process, we ask that you do the following: 

• Submit 1 hard copy and 1 electronic version of all the materials that comprise the 
review packet for each project. The hard copy version will allow us to assess whether 
the volume of material is appropriate, and will serve as a quick reference source 
should questions arise about the review packet. 
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Request for Materials for Expert Panel Reviews 



• If an item is not available in electronie format or if you would prefer the reviewers to 
have a hard copy version of a particular item (for example, a training binder with 
multiple tabs and color handouts included), please provide 3 additional hard copies of 
that item. 

• Each review packet should include a cover sheet (using the attached template) that 
lists all of the materials included in the packet. Electronic versions of the materials 
should be filed together in a project folder that is clearly labeled with the project 
name. Eeel free to use subfolders within the project folder if you think it will help 
reviewers organize the materials (for example, background/research base, meeting 
notes, training materials, etc.) in a more understandable way. You might also want to 
include the file name of each document referenced in question #5 of the cover sheet 
so that reviewers can easily sort through and match the documents on the cover sheet 
with the files they receive. 

• Please send your hard copy materials (bundled by project with a cover sheet on each), 
along with a CD containing separate folders for each of the sampled projects to: 

DIR 

Attn: 

[address] 

Materials are due no later than Oct 19, 2007. 

I have attached a copy of the quality scoring rubric so that you can see what the panelists 

will be looking for in the review process. As you begin to assemble the materials for 

panel review, we offer the following guidance: 

• With the exception of the cover sheet and the participant list, all materials provided in 
the review package should be materials that already exist. We do not expect you to 
create any new materials for the review process. 

• Materials included in the packet should focus on work conducted between summer 
2006 and summer 2007. Materials developed before summer 2006 can be included if 
they provide relevant contextual or background information. Materials developed 
after summer 2007 will be considered in the next round of panel reviews (for the 
2007-2008 program year), if that particular project is sampled again for review. 

• Make sure you have documented the “knowledge base” upon which the project was 
developed. As you can see in the attached rubric, this is a very important factor in the 
quality rating. Depending on the topic area and nature of the project, the knowledge 
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Request for Materials for Expert Panel Reviews 



base might include empirical research, laws and regulations, or professional 
wisdom^^. 

• Provide enough materials to allow reviewers to understand the project substantively, 
with a particular focus on the work conducted in the 2006-2007 program year. This 
does not mean sending everything ever done or developed in connection with the 
project. You want to provide reviewers with a sufficient basis for their ratings but at 
the same time, not overwhelm them with materials. 

• Focus more on the substantive materials than on those dealing with process or 
administrative issues alone. You do not need to include every e-mail message or set 
of meeting notes on a project unless it provides critical contextual information or 
speaks directly to the technical quality of the project. For example, if you are 
assembling materials about an annual forum or institute, you should not include 
meeting notes about conference logistics or attendee evaluation results, but you 
should include notes relating to the development of the agenda and selection of 
presenters. 

If you have any questions about what materials to include in the review packet or how to 

complete the cover sheet, please contact your site visitor. 



Compiling Lists of Project Participants 

For each project selected for the evaluation, a sample of participants will receive a short, 
web-based survey from the national evaluation. This is the reason we are asking each 
Regional Comprehensive Center (RCC) and each Content Center to compile a list of all 
state-level staff who have participated in each project sampled for the evaluation. Content 
Centers should also include all RCC-based staff, as well as all state-level staff, on their 
lists of project participants. Compile a separate list of participants for each project 
sampled, allowing duplication of names across lists. 

Defining “participants. ” “Project participants” include all those who have served on task 
forces and work groups associated with the project; state-level or RCC staff who have 
attended conferences, technical assistance retreats, consultations, and other meetings that 
are part of the project; and state-level or RCC staff who have received written materials 
or other resources disseminated under the project. 

If the Comprehensive Center collaborated with state-level staff to provide a service to 
others, and that collaboration was intended to build the capacity of the state in some way, 
then all of the state-level staff who collaborated with the Center on the project should be 



Professional wisdom is the judgment that individuals acquire through experience and is reflected through 
consensus views on decisions on implementation. Grover J. Whitehurst. Evidence-Based Education , 

2002 . 
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included on the list of project participants. These collaborators will be included in the 
pool of potential survey respondents. 

The same logic applies to Content Center projects that involve collaborations with RCC 
staff. Again, if the collaboration was intended to build the capacity of the RCC in some 
way, then all RCC staff who collaborated on the project with the Content Center should 
be included on the list of project participants and will be included in the pool of potential 
survey respondents. 

Defining “state-level. ” For the purposes of this evaluation, “state-level” participants in 
Regional Center or Content Center projects may include the following: 

• State education agency (SEA) employees and consultants 

• Employees of other state agencies, such as governors’ offices 

• Employees of intermediate education agencies^^ who assist schools on behalf of 
the state 

• Individuals who serve on school support teams as part of a statewide system of 
school support under NCEB, including local educators (school district 
administrators, principals, and teachers) 

• Individuals who serve on state-level work groups or task forces, including local 
educators 

Eocal educators who are not serving on school support teams and are not members of a 
state-level work group or task force should not be included in project participant lists. 

RCC staff. Content Centers should include all RCC staff who have participated in each 
project, in addition to state-level project participants. Participant lists for some Content 
Center projects may include no RCC staff, some may include both RCC and state-level 
staff, and some may be made up entirely of RCC staff, depending on the nature of the 
project. 

Contact information. For each project participant, please provide a name, title (if 
available), affdiation, email address, telephone number, and regular mail address. 
Because surveys will be administered via the web, with follow-up by regular mail and by 
telephone, all of this contact information is critical. Send a separate participant list for 



® Intermediate education agencies are usually established by state statute, but their governance structures 
and funding sources vary from state to state. Depending on the state, they are known as Area Education 
Agencies (AEA), Boards of Cooperative Educational Services (BOCES), Cooperative Education Service 
Agencies (CESA), County Offices of Education (COE), Education Service Centers/Cooperatives (ESC), 
Education Service Districts (ESD), Regional Education Service Agency (RESA), or Regional Education 
Service Centers (RESC). Association of Educational Service Agencies, “Questions Asked About 
Educational Service Agencies,” downloaded from http://www.aesa.us/QifeABro04.pdf on July 6, 2007. 
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each sampled project, following the template shown below (using either Word or Excel to 
create the file). 



Project name: Really Important Project 


Last 

name 


First 

name 


Title (if 
available) 


Affiliation/ 

Organization 


Email address 


Telephone 

nnmber 


Street 

address 


City 


State 


ZIP 


Doe 


John 


Title I 
Director 


State Department of 
Public Instmction 


idoe(S),sea.kl2.us 


(101) 555-1234 


99 State 
Street, 
Room 100 


State 

Capital 


ST 


10001 


Etc. 





















Participant lists for each project should be included with your sample review materials 
and sent to DIR no later than Oct 19, 2007. 

For additional information. If you have any questions about the participant lists, or if 
there are special circumstances pertaining to a project or to a group of participants that 

you would like to discuss, please contact at Policy Studies 

Associates. can be reached at (xxx) xxx-xxxx or by email at 

@ • 



Thank you for your continued cooperation. We look forward to receiving your materials. 
Sincerely, 

Vice President 

Decision Information Resources 
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National Evaluation of the Comprehensive Technical Assistance Centers 
Request for Materials for Expert Panel Review 



The projects below have been sampled from the Project Inventory Form that your Center completed for the most 
recent project year. These projects will be rated for quality by expert review panelists for the national evaluation 
of the Comprehensive Technical Assistance Centers. Surveys of participants in these projects will collect client 
ratings of the projects’ relevance and usefulness. 



Proiect 


Activities and Deliverables 











































In order to facilitate this process, for each project please provide us with: 

(1) Copies of the materials that together fairly represent each of these items. This should include: documents 
(electronically, wherever possible) leading up to, or resulting from, these efforts-including plans, agendas, 
meeting notes, handouts, presentations, follow-up memos, resolutions, instruments, assessments, summaries, 
syntheses, papers, reports and memoranda of agreement. In other words, we are asking for materials that fully 
describe these products and services for the reviewers and give them a sufficient basis for rating their quality. 

(2) A cover sheet (one per project) that provides background and contextual information about the project. (See 
attached page for specific questions to be answered on the cover sheet.) 

(3) A list of the state-level participants in each project, including names, affiliations, and contact information. 

Questions about what materials to provide and how to complete the cover sheet should be directed to your 

evaluation team liaison. 

These materials will be the basis upon which the reviewers will make their ratings. 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid 0MB control number. The valid 0MB control number for this data 
collection instrument is 1850 - 0823 . The time required to complete these worksheets is estimated to average four 
hours per response, including the time to review instructions, search existing data sources, gather the data needed, 
and respond to the questions. Send comments regarding this burden estimate or any other aspect of this collection 
of information, including suggestions for reducing the burden, to the U.S. Department of Education, Washington, 
DC 20202-4651. 
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Cover Sheet to Accompany Materials Sent to Review Panel for Sampled Projects 

Name of Comprehensive Center 

Name of person completing this form 

Telephone Email 



Information about Sampled Projects: 

Project Name 

Project Inventory # 



1 . Describe the purpose of this project and its intended outcome(s). 

2. Who is the customer/client? 



3. Describe the circumstances that led to the provision of this project. In this description, 
be sure to indicate whether it was initiated by the Center or requested by the client. 

4. Eist the activities and deliverables associated with this project, and describe how they 
relate to the overall project. 

5. Please provide a list of the materials and documents you are sending for the review 
panel. Eor each, describe the Comprehensive Center’s role and contribution to the 
materials associated with this project (i.e., the materials sent to reviewers for rating 
the quality and relevance of this project.) Please let us know if you used materials 
developed by another source and, if so, provide information about that source. 

6. What is the research basis for the product or service? Where appropriate, please 
provide a citation for the research or practice literature upon which it is based. 
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Sample Follow-Up Memo 



Dear [Center Director]: 

Members of the evaluation team met in Houston last week to look over the 
review materials submitted by the Centers. In our cursory review of the PCC 
projects, we had trouble locating certain items that were either described on the 
project cover sheet(s) or that one would expect to see in the project materials. 



Project Name 


Issue Identified 















If these items were included in your review package, please let us know where in 
the materials we can locate them so that we can make sure that the peer review 
panelists will be able to find them. 

If you didn’t intend to include them in the package, please let us know. 

If you would like to send new copies of these items or additional materials to 
address the issue(s) identified, please forward copies electronically to 

@ for hard copies to [name], DIR, [address] no later than 

November 8, 2007. 

If you have any questions, please feel free to contact me. 

Best, 

[name] 



B-26 





Appendix C 

Describing Center Operations 




Describing Center Operations 



This appendix contains a description of the process used to code the objectives within 
Center management plans, the PIFs, and the project cover sheets. This appendix also includes a 
description of the site visits, the open-ended interview protocols used for the 2007 site visits to 
CCs and RCCs, and the closed-ended prompts administered to the Centers in summer 2008. 
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Center Management Plan, Project Inventory, and Cover Sheet 
Coding and Intercoder Reliability 



Data from the Center management plans, projeet inventory forms (PIFs), and the projeet 
eover sheets were eoded by the evaluation team to build the sample frame and deseribe Center 
operations. Below is an explanation of the set of four proeedures used in the eoding proeess-(l) 
establish the eode sets, (2) train eoders, (3) eode management plans, PIFs, and eover sheets, (4) 
resolve eoding disagreements-as well as a summary of the level of intereoder reliability aehieved 
through this proeess. 



Establish the Code Sets 

Based on initial review of the projeet eover sheets, the PIFs, and the other materials from 
the Centers (sueh as the Center management plans), the evaluation team established eode sets to 
use during the eoding proeess. One set of eodes was used to eode the objeetives stated in the 
Center management plans and the topies of the projeets on the PIFs. Another set of eodes was 
used to eode the projeet eover sheets. 



Center Management Plan and PIF Code Set 

The team developed eodes to summarize the primary topies in whieh Centers were doing 
their work. Review of Center materials resulted in the identilieation of eategories of distinet 
domains or topies. These 22 speeifie topie eodes are deseribed below in terms of the substantive 
knowledge base the evaluation team determined would be eovered by that topie. In the eoding 
proeess, this set of possible eodes was used to eode the topie areas for the Center objeetives as 
stated in their management plans as well as to eode the topie area of eaeh projeet that Centers 
submitted on their PIFs. 

■ Components of effective systems of support-state, district, school. This eategory 
ineluded: (1) Design of state systems to meetNCLB requirements for statewide 
systems of support for distriets and sehools in eorreetive aetion or identified as in 
need of improvement [See. 1 1 17(a)]; (2) Interventions for sehools or distriets in 
need of improvement or eorreetive aetion; (3) For sehools. Centers' projeets 
addressed a range of options for sehool leadership and organization, professional 
development and elassroom praetiee, and development and implementation of 
sehool improvement plans. For distriets. Centers' projeets typieally addressed 
distriet leadership and development as well as implementation of distriet 
improvement plans. 



Discussion in the report refers primarily to the 22 specific substantive topics. However, as a small number of 
sampled projects fell into the category of “other” topic area, description of all the projects in the study sample by 
topic as displayed in Exhibit 2.4 includes the topic area of “other” as the 23'“* category. 
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■ Data use/Data-driven decisionmaking. Use of data for educational program 
improvement. 

■ Formative assessment. Guidelines for implementing formative or benchmark 
assessment, comprehensive formative assessment systems. 

■ Reading. Alignment of curriculum, instruction, and assessments with state 
standards in reading; teacher professional development in reading (not specific to 
adolescent literacy). 

■ Adolescent literacy. Policy and practices relevant to literacy at middle school and 
high school levels. 

■ Mathematics. Alignment of curriculum, instruction, and assessments with state 
standards in mathematics as well as teacher professional development in 
mathematics. 

■ Dropout prevention. Policies and practices specifically designed to keep high 
school students in school through graduation. 

■ High school redesign/reform. High school renewal, school organization, 
curriculum, and instructional practice. 

■ Transition to high school. Policies and practices designed to improve the 
preparation of middle-school students for success in high school. 

■ Special education-curriculum, instruction, and professional development. 
Curriculum, instruction, and professional development for service to students with 
disabilities. 

■ Special education-assessment. Implementation and interpretation of assessments 
for students with disabilities, including alternate assessments. 

■ English Language Learners. Curriculum, instruction, and professional 
development for service to ELL students; Title III improvement plans. This 
category also includes determination of ELLs’ achievement of AYP through 
assessment of Annual Measurable Achievement Objectives; implementation and 
interpretation of ELL assessments. 

■ Highly Qualified Teacher provisions of NCLB. State plans and policies required 
in connection with the HQT provisions of NCLB, including HOUSSE and 
equitable distribution. 

■ Teacher preparation and induction. State policies and practices for pre-service 
teacher preparation and induction of novice teachers as well as activities targeting 
teacher retention. 

■ Teacher professional development. State policies and local practices for the 
planning and delivery of professional development for teachers (not specific to 
reading, mathematics, high schools, ELLs, or special education). Activities in the 
area of effective teaching are included in this category. 

■ Supplemental Educational Services (SES). Application process to select vendors, 
monitoring services, and evaluating performance of providers. 

■ Response to Intervention (RtL). Rtl implementation, moving Rtl into general 
education, Rtl as a strategy for school improvement. 

■ Migrant education. Statutorily required state Comprehensive Needs Assessment 
(CNA) process for migrant education. 
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■ Indian/Native American education. Policies and practices to support the 
edueational needs of Ameriean Indians, Alaska Natives, and Native Hawaiians; 
ineludes support of and eomplianee with ESEA Title VIE 

■ Data management compliance. Title I eomplianee, EDEN/EDEacts reporting 
requirements. 

■ Assessment design. Developing new assessments for use by elients; ineludes 
elassroom, benehmark, and state grade-level assessments for federal 
accountability; activities may inelude design, item development, validation, 
reliability testing, alignment studies. 

■ Parent involvement. Title I and other parent involvement aetivities. 



Cover Sheet Code Sets 

The team developed eodes to deseribe projeet aetivities and resourees, eontributions CCs 
made to RCC projeets, and contributions RCCs made to CC projeets. The development of these 
eodes was informed by the team’s review of the Center management plans, the transcripts of the 
interviews eondueted in 2007, and a preliminary review of all projeet eover sheets. The eode sets 
used to eode the projeet eover sheets are shown below along with the eriteria used to assign that 
eode. 

Types of Project Activities and Resources 

m Engagement of participants in project planning was eoded for projeets that 

offered some or all of their partieipants the opportunity to take part in designing 
the projeet. This involvement went beyond merely expressing a need for teohnieal 
assistance; rather, it ineluded participant involvement in shaping the speeifie 
assistanee that was delivered (who did what, when, and why in technieal 
assistanee) at various stages of teohnieal assistanee delivery). 

Projeet planning was different from “ongoing oonsultation and follow-up,” though 
the same project may have had both components. “Ongoing consultation and 
follow-up” occurred when the Center provided servioe to elients on multiple 
oooasions, whereas “engagement of partieipants in projeet planning” was speeifie 
to the Center working with the client to plan later serviee. 

■ Research collections or syntheses were researeh-based materials distributed to 
partieipants in a projeet. In some instanees these produets were developed by the 
Center conducting the project, but in others they were existing researeh materials 
that the Center eollected and distributed. 

■ Training events were designed to impart skills to partieipants, equipping them to 
carry out a particular program or strategy. The eontent of trainings was diserete 
and speeialized, and partieipants attended trainings expeeting to learn something 
speeifie and be able to do something differently afterward. Partieipants and 
trainers assumed that the primary purpose of the training was for the partieipant to 
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learn something from the trainer. Projects with multiple trainings to the same 
participants over time were also coded as “ongoing consultation and follow-up.” 

■ Conferences featured presentations of information to audiences from multiple 
speakers or discussants representing a range of perspectives. They focused on 
informing the participants, not training them, and they were single events rather 
than series of meetings. The conference scope was broader than trainings, in that 
presentations or discussions in a single conference could have addressed multiple 
issues, programs, or solutions; whereas a training focused on a single strategy or 
program. Conference participants varied in their type, interest, and expectation for 
what they wanted to learn at the conference. Conferences also included symposia 
and forums in which participants came at common questions with varying 
perspectives. 

■ Support for development of a formal plan to implement a program or policy 

referred to instances in which Center technical assistance supported work by 
participants that led to a plan for implementing a program or policy. A formal 
state plan codified activities that were expected to occur in the state in order to 
serve state purposes with the intention to guide individuals and units toward a 
common goal. Formal plans may have been required by the federal or state 
government, or may have been initiated within the SEA to address a specific 
problem. This activity did not include service plans. 

■ Task force meetings and work were engagements over multiple interactions in 
which a Center provided technical assistance to a work group who were 
themselves constituted as a task force to address a state problem. Center support 
of a task force meant that the Center was a regular participant in task force 
meetings, and Center support ranged from setting the agenda and facilitating 
meetings to providing input upon request. A project was coded as “task force 
meetings and work,” when there was a task force that was trying to accomplish 
some purpose — not just to plan or follow up on Center technical assistance, but to 
do something for its state (or, occasionally, a group of states). Center participation 
in a task force was also coded “ongoing consultation and follow-up” if the Center 
provided assistance on an ongoing basis over time. If the Center was working 
with the task force to plan service, the project was also coded “engagement of 
participants in project planning.” 

■ Ongoing consultation and follow-up was either a series of consultations (i.e., 
fulfilling repeated requests from participants for technical assistance) or sustained 
follow-up with individuals who had participated in some other kind of assistance 
(e.g., a conference or training). Ongoing consultation occurred when there were 
multiple service contacts over time, all related to the project goal or topic. The 
interactions focused on a client question or need. This code did not include 
projects with one-time contacts, such as projects that provided a large conference 
with no follow-up. It also did not include the production of publications that are 
not accompanied with follow-up assistance directly to the client. 



C-5 




RCC and CC Contributions to RCC and CC Projects 



■ RCC projects with CC contribution were projeets in whieh the CC explieitly 
contributed a service or resource used by the RCC or its clients as part of RCC 
project activities. A CC could contribute to an RCC project in any of three ways: 

■ CC provided materials used in this RCC project. The RCC must have 
used CC materials in specific delivery activities or in planning. 

■ CC provided advice to the RCC on this project. The RCC consulted or 
collaborated with the CC when making decisions about what or how to 
deliver assistance to project participants. Participants may or may not have 
had direct interaction with the CC or CC materials. 

■ CC delivered technical assistance to RCC project participants. The CC 
provided service directly to the participants, whether face-to-face, by 
phone, or virtually, regardless of what role RCC staff had in this RCC 
project activity. 

■ CC projects with RCC contribution were projects in which the RCC explicitly 
contributed assistance used by the CC or its clients as part of CC project 
activities. An RCC could contribute to a CC project in either of two ways: 

■ RCC recruited participants or brokered service. The RCC connected the 
CC and its clients for the purpose of participation in the project. The RCC 
may have been driven by the CC’s need for participants who could benefit 
from a specific project, or by a client’s need for specific services or 
products that a CC could deliver. 

■ RCC delivered technical assistance to CC project participants. The RCC 
provided service directly to participants, whether face to face, by phone, or 
virtually, regardless of what role CC staff had in this CC project activity. 



Train Coders 

All coders were senior members of the evaluation team and were trained to use the 
established code sets. As part of the training, coders discussed each of the codes in the code set 
to ensure all coders had a shared understanding of their meaning. They also coded sample items 
and reviewed, coded, and discussed the final codes using representative sample text. The 
evaluation team also used the discussions in the coder training to verily that the established 
codes were clear and well understood. 
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Code Center Management Plans, PIFs, and Cover Sheets 



All of the objectives in the Center management plans, the PIFs, and cover sheets were 
independently coded by two coders. For the Center management plans, each coder reviewed the 
objectives section of the plan and determined the topic(s) in which the Center indicated they 
planned to work. For the PIFs, each coder reviewed the project description and assigned the 
project subject code for each of the 364 projects. The cover sheets were reviewed by each coder 
to categorize the project information submitted by Centers (i.e., project purpose, customer/client, 
project activities and deliverables, roles and contributions of Centers, research basis for products 
and services) and to assign codes according to the coding criteria described earlier (see p. C-5) 
for identifying the various types of project activities and resources and the ways Centers 
contributed to projects. 



Resolve Coding Disagreements 

Members of the evaluation team compared the coding worksheets from each coder to 
determine the final coded responses for each item. In cases where the coders agreed, the final 
response was clear. In cases where the first and second coders did not arrive at the same code, 
they met to review evidence for the codes and choose a final code. If the first and second coder 
were unable to reconcile their codes, a third senior study team member worked with the coders to 
determine the final code for that item. The product of this step was the final codes for the topic 
areas included in the Center management plan objectives, the topic areas for the 364 projects on 
the PIFs, and the three areas described above for the project cover sheets. 



Analysis of Intercoder Reliability 

The evaluation team calculated the percent agreement between the first and second coder 
as well as Cohen’s Kappa analysis. The percent agreement calculation used the number of 
agreements divided by total number of possible responses. The Cohen’s Kappa calculation used 
the number observed to agree minus the number expected to agree by chance divided by the 
number of items minus the number expected to agree. 

For the topic area assignment of the Center objectives stated on their management plans, 
the first and second coder agreed on 452 of 462 possible topic area assignments (98 percent 
agreement across the 22 topic areas for each of 21 Centers). The overall Cohen’s Kappa between 
the two coders was .96 for this coding activity. 

For the PIFs, the first and second coder agreed 87 percent of the time (on 318 of the 364 
subject codes) and the overall Cohen’s Kappa between the two coders was .92. Chapter 3 of the 
report used this information at an aggregated level-work done by subject, by Center-showing 
whether or not each Center had at least one project listed on its PIF for each of the 22 subject 
areas. To determine the intercoder reliability at this aggregate level, the evaluation team 
examined the final 462 cell matrix (22 topic areas for each of 21 Centers) showing the subject 
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areas in which each Center reported projects on its PIF against the similar matrices for the first 
and second coder. When viewed at this level, the intercoder reliability was even higher. Of the 
462 possible yes/no designations, 98 percent were the same for the first and second coder, with a 
Cohen’s Kappa of .95. 

For the project cover sheets, the first and second coder agreed on 1,572 of 1,830 possible 
codes (122 projects with a possible 15 codes for each project) or 86 percent of the time. The 
overall Cohen’s Kappa between the two coders was .89. 



Site Visit Interviews 

Two rounds of site visits were conducted with the Center Director and other senior 
Center staff to gather information about the Center operations during the 2006-07 program year 
through structured interviews. The first round of interviews in summer 2007 were based on 
open-ended questions posed to RCC and CC staff about Center organization, major areas of 
focus, communication with client organizations, sources of knowledge used, approaches taken in 
quality assurance, modes of delivering technical assistance, and barriers they encountered (see 
page C-9 for the CC protocol and page C-12 for the RCC staff interview protocol). The protocol 
for the second interviews included closed-ended, binary questions to follow up specifically about 
activities that had occurred in 2006-07, topics that had been addressed, and work requests that 
had fallen outside their planned scope of work. Additional questions targeted the strategies that 
Centers had used for planning work and engaging clients, the sources for content expertise, and 
sources they used for vetting their products and services (see protocol on page C-15). Responses 
to the second interview prompts are reported in chapter 3 as part of the description of Center 
operations. 



C-8 




CC Staff Site Visit Interview Protocol 



National Evaluation of the 
Comprehensive Technical Assistance Centers 

Content Center (CC) Staff 
Site Visit Interview Protocol 



1 . Tell me about your professional background and areas of expertise. What is your primary 
role within this Center? Describe the tasks you might perform or decisions you might 
make during a typical one- or two-week period? 

2. Give us an overview of your Center’s internal structure and operations: 

—How are staff and consultants assigned, and work organized (e.g., by Regional Center, 
functional specialization)? 

—How is communication initiated and supported between the Center and its clients? 
—How are decisions made regarding the provision of services (e.g., the nature and 
duration of services)? 

3. Describe your Center’s relationship with your larger host organization. What are the 
advantages and disadvantages to the Center of being housed in the host organization? To 
what extent can you access expertise and other resources from your host organization to 
support the work of the Content Center? 

4. Describe your Center’s relationship with the Regional Centers (e.g., the frequency, 
methods, and content of communication). Do you interact with some Regional Centers 
more than others? 

5. Describe your Center’s relationship with other Content Centers (e.g., the frequency, 
methods, and content of communication). Do you interact with some Content Centers 
more than others? How do you work together with other Content Centers to define roles 
and responsibilities when there is overlap in your areas of expertise or organizational 
capabilities? 

6. Can you give us some specific examples of ways in which other Content Centers or 
Regional Centers have helped or supported your Center to achieve a goal or objective? 

7. Describe the major focus areas or initiatives your Center has addressed or engaged in 
during the past year. How did you determine these specific priorities? To what extent do 
you find it necessary to conduct formal needs assessments? [If applicable] How are these 
done? 

8. Which of your current initiatives do you consider to be the largest or most significant 
(e.g., in terms of importance, amount of resources allocated, etc.)? 

Prompts: 
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CC Staff Site Visit Interview Protocol 



—How and why did it begin? 

—What products and services have you provided? 

—What are the goals and objectives; how much progress have you made to date? 
—What obstacles and challenges have you encountered? 

9. To what extent do you plan and initiate the delivery of products and services to the 
Regional Centers versus respond to specific Regional Center requests? 

10. To what extent have Regional Centers expressed needs that are different from your 
Center’s priorities? 

11. How and to what extent do Regional Centers vary in the nature (e.g., topics); frequency, 
and type (e.g., seminars, requests for materials, etc.) of assistance they request? 

12. Other than the Regional Centers, whom do you serve? What needs and priorities have 
these other clients expressed? To what extent do you work directly with states? [If 
applicable] What products and services do you typically provide to states, and which 
offices do you work with? To what extent do you work with the U.S. Department of 
Education? 

13. To what extent do needs and priorities overlap among different clients? Describe any 
strategies you may have developed for maximizing your Center’s efficiency in 
responding to similar needs across multiple clients? 

14. To what extent do your clients request products or services that address special 
education-related needs or questions? 

15. What technical assistance methods do you rely on the most as you try to meet priority 
needs? Are there unique drawbacks or benefits associated with employing specific 
techniques? 

Prompts: 

—facilitating inter-organizational cooperation and collaboration (with whom?) 

—identifying and disseminating documents and other products 

—developing new products and plans 

—assisting clients to implement new plans or systems 

—hosting workshops, seminars, and other training events 

—identifying and using scientifically based evidence 

16. Where and how do you get the substantive knowledge that you use in your products and 
services? How do you decide what to use and what not to use in your work with clients? 
What quality standards or guidelines, if any, do you apply? 

17. Where does your Center turn for help when, for example: (1) you experience challenges 
or encounter obstacles that limit your effectiveness, or (2) you need to increase 
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CC Staff Site Visit Interview Protocol 



institutional capacity (e.g., develop new expertise or sharpen your skills in a particular 
area)? [If applicable]; 

—Give specific examples that illustrate how others have helped your Center. 
—Indicate if you seek help on an as-needed basis, or whether there are regularly 
scheduled workshops and trainings? 

18. What does your Center do best? What kinds of needs are you best equipped to address? 
As you consider your Center’s many objectives, which ones do you believe you have met 
most satisfactorily? 

19. In what ways, if any, has your Center changed in recent years (e.g., over the past one to 
two years)? Describe, for example, new developments in how you organize and operate, 
hire and deploy staff and consultants, use technology, allocate resources, or in how you 
otherwise plan, design, or deliver technical assistance. What motivated these changes and 
what kinds of effects or outcomes have resulted? 

20. In what ways, if any, has the context within which you work changed in recent years 
(e.g., over the past one to two years)? Describe, for example, shifts in needs and 
priorities, new developments in your field, or changes in rules or regulations that affect 
your Center and the work that you do. What impact has this had on your Center, and on 
the nature and effectiveness of the work that you do? 

21. With which Regional Centers do you work most frequently? How does your organization 
assess whether it is helping to build capacity within these Centers? 

22. What factors and conditions obstruct or hinder your work in significant ways? How have 
you tried to address them and with what level of success? 

23. The quality and effectiveness of technical assistance is dependent on many factors. Based 
on your experiences, what recommendations or suggestions would you make to the U.S. 
Department of Education, state policymakers, your clients, or others, regarding ways to 
improve or support the work of the current Content Centers? 
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RCC Staff Site Visit Interview Protoeol 



National Evaluation of the 
Comprehensive Technical Assistance Centers 

Regional Center (RCC) Staff 
Site Visit Interview Protocol 



1 . Tell me about your professional baekground and areas of expertise. What is your primary 
role within this organization? Deseribe the tasks you might perform or deeisions you might 
make during a typieal one- or two-week period? 

2. Give us an overview of your Center’s internal structure and operations; 

—How are staff and consultants assigned? 

—How is communication initiated and supported between the Center and its clients? 

—How are decisions made regarding the provision of services (e.g., the nature and 

duration of services)? 

—How do you organize your work (e.g., by state, functional specialization)? 

3. Describe your Center’s relationship with your larger host organization. What are the 
advantages and disadvantages to the Center of being housed in the host organization? To 
what extent can you access expertise and other resources from your host organization to 
support the work of the Regional Center? 

4. Describe the major focus areas that your Center has addressed or engaged in during the past 
year. How did you determine these specific priorities? To what extent do you find it 
necessary to conduct formal needs assessments? [If applicable] How are these done? 

5. Which focus area do you consider to be the largest or most significant (in terms of 
importance, amount of resources allocated, etc.)? Tell us about it. 

Prompts: 

—How and why did it begin? 

—What products and services have you provided? 

—What are the goals and objectives, and how much progress have you made to date? 

—What obstacles and challenges have you encountered? 

6. In your work with states, what aspects of NCLB is your Center most often called upon to 
address within your region? Is there much variation among SEAs in this regard? Describe the 
specific kinds of help that are most often requested. 

7. To what extent have states expressed different or additional needs to those described above? 

8. Describe your relationships with each of the states in your region (e.g., the frequency, 
duration, and purpose of your interactions with them). To what extent do these interactions 
help to shape your Center’s priorities and policies? 
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9. With whieh offiees or departments within states do you work most frequently? Does this vary 
among the states in your region? Is there a eoordinating body within eaeh state or among the 
states that facilitates communication with your Center? Please name the senior managers 
with whom you coordinate your work in each state. 

10. Other than states, whom do you serve? What priorities have emerged among these other 
clients? How often do you serve these nonstate clients? Are you able to be as responsive to 
them as you would like? 

11. Describe your relationships with each of the Content Centers (e.g., the frequency, duration, 
and purpose of your interactions with them). To what extent do these interactions help to 
shape your Center’s priorities and policies? 

12. Consider the full range of needs and priorities that this Center, the states, and other clients 
within the region have identified. To what extent do needs and priorities overlap among 
different clients? Describe any strategies you may have developed for maximizing your 
organization’s efficiency in responding to similar needs across multiple clients. 

13. What technical assistance methods do you rely on the most as you try to meet priority needs? 
Are there particular drawbacks or benefits associated with employing specific techniques? 

Prompts: 

—facilitating inter-organizational cooperation and collaboration (with whom?) 

—identifying and disseminating documents and other products 

—developing new products and plans 

—assisting clients to implement new plans or systems 

—hosting workshops, seminars, and other training events 

—identifying and using scientifically based evidence 

14. Where and how do you get the substantive knowledge that you use in your products and 
services? How do you decide what to use and what not to use in your work with clients? 

What quality standards or guidelines, if any, do you apply? 

15. Where does your Center turn for help when, for example: (1) you experience challenges or 
encounter obstacles that limit your effectiveness, or (2) you need to increase institutional 
capacity (e.g., develop new expertise or sharpen your skills in a particular area)? [If 
applicable]: 

—Give specific examples that illustrate how the Content Centers and other 
organizations have helped your Center. 

—Indicate if you seek help on an as-needed basis, or whether there are regularly 
scheduled workshops and trainings. 

16. Describe your organization’s relationship with the Content Centers and other Regional 
Centers (e.g., the frequency, methods, and content of communication between organizations). 
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RCC Staff Site Visit Interview Protoeol 



Do you interact with some Content Centers or Regional Centers more than others? Can you 
give us some specific examples of ways in which other Centers have helped to increase your 
capacity or effectiveness, or ways your organization has supported the work of other 
Centers? 

17. What does your Center do best? What kinds of needs are you best equipped to address? As 
you consider your organization’s many objectives, which ones do you believe you have met 
most satisfactorily? 

18. In what ways, if any, has your organization changed in recent years (e.g., over the past one 
to two years)? Describe, for example, new developments in how you organize and operate, 
hire and deploy staff and consultants, use technology, allocate resources, or in how you 
otherwise plan, design, or deliver technical assistance. What motivated these changes and 
what kinds of effects or outcomes have resulted? 

19. In what ways, if any, has all or parts of the region in which you work changed in the past two 
years? Describe, for example, any political, social, institutional, legal, economic, or 
demographic changes that have affected the education system in significant ways. What 
impact has this had on your Center, and on the nature and effectiveness of the work that you 
do? 

20. To what extent have the states in your region changed (e.g., structurally or operationally) as a 
direct result of your Center’s efforts? [If applicable] Give specific examples of changes that 
have occurred. 

21 . How does your Center assess whether it is helping to build capacity within the states in your 
region? If change has been difficult to achieve, why do you think that is the case? 

22. What factors and conditions obstruct or hinder your work in significant ways? How have you 
tried to address them and with what level of success? 

23. The quality and effectiveness of technical assistance is dependent on many factors. What 
recommendations or suggestions would you make to the U.S. Department of Education, state 
policymakers, your clients, or others, regarding ways to improve or support the work of the 
current Regional Comprehensive Centers? 
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Interview Prompts 



National Evaluation of the Comprehensive Centers 

Interview Prompts 

On-site, the site visitor asked the Center to respond to the items shown in the following 
tables, with respeet to Center objectives and operations in 2006-07. 

Table 1. Management Plan Objectives That Address Particular Topics 



Topics 


In management 
plan 
(Y/N) 


a. 


Components of Effective Systems of Support -State, District, School 




b. 


Data Use / Data-driven decision making 




c. 


Formative Assessment 




d. 


Reading 




e. 


Adolescent Literacy 




f. 


Mathematics 




g- 


Dropout Prevention 




h. 


High School Redesign/Reform 




i. Transition to High School 


j- 


Special Education - curriculum, instruction and professional 
development 




k. 


Special Education - assessment 




1. English Language Learners 


m. 


Highly Qualified Teacher provisions of NCLB 




n. 


Teacher preparation and induction 




o. 


Teacher professional development 




P- 


Supplemental Educational Services (SES) 




q- 


Response to Intervention (Rtl) 




r. 


Migrant education 




s. 


Indian/Native American education 




t. 


Data Management Compliance 




u. 


Assessment Design 




V. 


Parent Involvement 




w. 


Other 
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Table 2. Center Strategies for Planning Work 

Center strategies: (Y/N) 

a. Needs assessment through survey of client staff 

b. Needs assessment through analysis of state performance data or 
accountability plans 

c. Needs assessment through analysis of state and federal policy 
context 

d. Needs assessment through meetings or other communication with 
clients 

e. Center included client input when writing the original grant proposal 
or cooperative agreement with ED 

f. Center includes client input when writing annual management/work 

plan 

g. Center communicates directly with the director of client 
organizations to plan the Center’s work 

h. Center builds on relationships with clients that existed prior to the 

Center’s grant award. 

i. Center refers to parameters set by NCLB or the Center’s 

authorizing legislation when negotiating with clients to plan its work 



Table 3. Center Strategies for Building Client Engagement and Buy*ln 

Center strategies: (Y/N) 

a. Work on topics outside planned scope of work 

b. Work on tasks outside planned scope of work 

c. Work with primary service recipients outside planned scope of work 

d. Sponsor large events to make contact with many clients 

e. Form work groups within client organizations that bring together staff 
from multiple departments/divisions to discuss service needs and 
delivery 

f. Form work groups across client organizations to discuss service 

needs and delivery 

g. Offer service to support existing client programs/projects/policies 

h. Hires former employees of client organizations (SEAs or RCCs) 

i. Involves clients in making hiring decisions about Center staff 

j. Has staff positions designated for serving as an in-house liaison to a 
specific client organization (SEA or RCC), maintaining office space 
within the client’s building 
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Table 4. Center Strategies for Managing Work Requests Falling Outside Their 
Planned Scope 



Center strategies: (Y/N) 


a. 


Center preempted requests for work outside its planned 
scope by having introductory meetings with clients in 
order to define the parameters of work it would do 


b. 


Center refers to parameters set by NCLB or the Center’s 
authorizing legislation when negotiating with clients 


c. 


Center agrees to work outside its planned scope in order 
to build a working relationship with clients 


d. 


Center has a staff member and process for ensuring that 
Center work is within its work plan 


e. 


Center submits a revised work plan for ED approval before 
responding to requests outside its plan 


f. 


Center has declined to provide a service because it was 
outside of its planned scope 



Table 5. Sources of Content Expertise 



Source 


(Y/N) 


a. 


Center staff themselves (lead organization and/or subcontractors) 




b. 


Other staff in the lead organization, not formally employed by the 
Center 




c. 


Other staff in subcontractor organization(s) 




d. 


Content Center(s) 




e. 


Universities 




f. 


State education agencies 




g- 


What Works Clearinghouse 




h. 


Professional associations (e.g., AERA, CCSSO) 




i. Consulting firms or private contractors 


j- 


Other regional Center(s) 




k. 


Other federally funded technical assistance providers (not part of the 
Center) 




1. Regional labs 
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Table 6. Sources for Vetting for Products and Services Prior to Delivery 



Source 


(Y/N) 


a. 


Internal quality control by Center staff (lead organization and/or 
subcontractors) 




b. 


Other staff in the lead organization, not formally employed by the 
Center 




c. 


Other staff in subcontractor organization(s) 




d. 


U.S. Department of Education 




e. 


Outside experts retained to review drafts 




f. 


Content Center(s) 





Table 6. Center Strategies for Managing Work Requests Falling Outside Their 
Planned Scope 



Center strategies: 


From year 1 
site visit 
(Y/N) 


a. 


Center refers to parameters set by NCLB or the Center’s authorizing 
legislation when negotiating with clients 




b. 


Center agrees to work outside its planned scope in order to build a 
working relationship with clients 




c. 


Center has a staff member and process for ensuring that Center work 
is within its work plan 




d. 


Works with primary service recipients outside planned scope of work 




e. 


Center has declined to provide a service because it was outside of its 
planned scope 
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Appendix D 

Survey of Senior State Managers 




Survey of Senior State Managers 



This appendix section presents the survey that was administered to Senior State Managers 
serving as RCC’s main points of contacts (see page D-2). The survey was developed to obtain 
information from state managers about state-level needs and priorities for technical assistance, 
state perspectives of Center technical assistance, and comparative judgment of Center assistance 
in relation to assistance available through other sources. 



D-1 




State Manager Survey 



Comprehensive Center Evaluation 
Senior State Manager Survey 

This survey is designed to gather data on technical assistance received by your state from 
a variety of external sources, including federally funded technical assistance centers, colleges 
and universities, professional associations, colleagues in other states, consulting firms, and 
private contractors. 

In particular, the survey asks about assistance your state has received from the federally 
supported network of Comprehensive Technical Assistance Centers, which includes in your 
region the following Regional Comprehensive Center; 



This Regional Center often collaborates with one or more Content Centers, which may also work 
directly with states: 

■ The Assessment and Accountability Comprehensive Center (AACC), which is 
directed by Dr. Stanley Rabinowitz at WestEd in San Francisco, California. 

■ The National High School Center (NHSC), which is directed by Dr. Joe Harris 
at the American Institutes of Research in Washington, DC. 

■ The Center on Innovation and Improvement (CII), which is directed by Dr. 
Sam Redding at the Academic Development Institute, in Lincoln, Illinois. 

■ The Center on Instruction (COI), which is directed by Ms. Angela Penfold at 
RMC Research Corporation in Portsmouth, NH. 

■ The National Comprehensive Center for Teacher Quality (NCCTQ), which is 
directed by Dr. Sabrina Laine at Learning Point Associates in Naperville, Illinois. 

Questions about the Comprehensive Center network in this survey refer to all of the 
Comprehensive Centers that have served your state, including the Regional Center and any 
Content Centers with which your state has worked. 

The term “technical assistance,” as used in this survey, includes print and electronic 
information and other resources, consultation, advice, facilitation, and training. 
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State Manager Survey 



Per the Edueation Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to 
this data collection will be used only for statistical purposes. The reports prepared for this study 
will summarize findings across the sample and will not associate responses with a specific 
organization or individual. We will not provide information that identifies you or your 
organization to anyone outside the study team, except as required by law. 



Please return this survey using the self-addressed, stamped envelope enclosed 
to [name] of Policy Studies Associates (PSA) at: 

[address] 

@ 



Thank you for your feedback! 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid OMB control number. The valid OMB control number for this survey is 1850- 
0823. The time required to complete this survey is estimated to average 20 minutes per response, including the time 
to review instructions, search existing data sources, gather the data needed, and respond to the survey questions. 
Send comments regarding this burden estimate or any other aspect of this collection of information, including 
suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 
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Priorities for Technicai Assistance 

1 . When your state requests teehnieal assistanee from outside sources, what are its 

priorities? To what extent is each of the following state responsibilities related to NCLB 
implementation a priority for the technical assistance the state requests? (Select one 
response in each row.) 

Note: Please consider the technical assistance that your state has sought from all 

sources, including federally funded technical assistance centers, colleges and 
universities, professional associations, colleagues in other states, consulting 
firms, and private contractors. 



State Responsibilities 


Major 

priority 


Moderate 

priority 


Minor 

priority 


Not at all a 
priority 


a. Formulating or refining state policies to 
respond to NCLB requirements 


4 


3 


2 


1 


b. Building or managing a statewide system 
of support for districts and schools 
identified for improvement under NCLB 


4 


3 


2 


1 


c. Training or managing the state-level staff 
or school support teams who provide 
support to districts and schools identified 
for improvement under NCLB 


4 


3 


2 


1 


d. Designing or implementing state 

assessment and accountability systems 


4 


3 


2 


1 


e. Aligning state accountability systems with 
NCLB accountability systems 


4 


3 


2 


1 


f. Supporting use of assessment data by 
schools and districts 


4 


3 


2 


1 


g. Disseminating information on 

scientifically based research to districts 
and schools 


4 


3 


2 


1 


h. Identifying and/or developing programs 
or models that address district and/or 
school needs 


4 


3 


2 


1 


i. Providing training and other professional 
development to local educators in 
academic subjects (reading language arts, 
mathematics, science) 


4 


3 


2 


1 


j. Monitoring compliance with NCLB 
requirements in districts and schools 


4 


3 


2 


1 


k. Communicating with the public about 
NCLB requirements or report cards 


4 


3 


2 


1 


1. Other (Specify: ) 


4 


3 


2 


1 
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State Manager Survey 



Sources of Technical Assistance 

2. During the 2006-07 sehool year (beginning July 2006 and ending June 2007), to what 
extent has your state relied on eaeh of the following sourees of teehnieal assistanee? 

(Select one response in each row.) 



Sources of Technical Assistance 


One of the 
state’s 
most 

important 

resources 


To a great 
extent, but 
not one of 
the state’s 
most 

important 

resources 


To a 

moderate 

extent 


Minimally 


No contact 


a. U.S. Department of Education 
(Specify office: 

) 


4 


3 


2 


1 


0 


b. Professional associations (e.g., 
CCSSO, ASCD) 


4 


3 


2 


1 


0 


e. Colleges and universities 


4 


3 


2 


1 


0 


d. Consulting firms or private 
contractors 


4 


3 


2 


1 


0 


e. Your counterparts in other SEAs 


4 


3 


2 


1 


0 


f. Comprehensive Center network 


4 


3 


2 


1 


0 


g. Regional Educational Eaboratory 


4 


3 


2 


1 


0 


h. Other federally funded technical 
assistance providers (Specify: 

) 


4 


3 


2 


1 


0 


i. Other (Specify: 

) 


4 


3 


2 


1 


0 
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State Manager Survey 



3. For what purpose(s) does your state seek assistanee from eaeh of the following sourees? 

(Select all that apply for each source.) 

a. U.S. Department of Edueation (Specify office(s): ) 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

b. Professional associations (e.g., CCSSO, ASCD) 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

c. Colleges and universities 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

d. Consulting firms or private contractors 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 
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e. Counterparts in other SEAs 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

f. Comprehensive Center network 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

g. Regional Educational Eaboratory 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

h. Other federally funded technical assistance providers (Specify: 
) 



□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 

i. Other (Specify: ) 

□ To gather information or to keep current with new ideas 

□ To plan the initial steps in solving a problem 

□ To complete tasks that the state could do itself if it had more staff or resources 

□ To complete tasks for which the state lacks expertise 

□ To develop the skills of SEA or intermediate education agency staff 

□ To work directly with districts and schools 

□ Not applicable; our state doesn ’t seek assistance from this source 
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Are there other purposes for whieh your state seeks assistanee from outside sourees, not 
mentioned in Question 3? To whieh sourees do you turn for these purposes? Please 
deseribe briefly below. 




State Manager Survey 



Technical Assistance Received from the Comprehensive Center Network 

5. During the 2006-07 sehool year (beginning July 2006 and ending June 2007), with 

whieh responsibilities related to NCLB implementation did your state reeeive assistanee 
from the Comprehensive Center network (your Regional Center and the Content Centers 
with whom your state has worked)? (Select one response in each row.) 



Our state received assistance 
from the Comprehensive 
Center network with the 
following responsibilities: 


Major 

assistance 


Moderate 

assistance 


Minimal 

assistance 


No 

assistance 
at all 


NA, 

technical 
assistance 
not sought 
by the 
SEA 


NA, this 
task is not 
relevant to 
my unit’s 
work 


a. Formulating or refining 
state policies to respond to 
NCLB requirements 


4 


3 


2 


1 


Cl 95 


CI98 


b. Building or managing a 
statewide system of 
support for districts and 
schools identified for 
improvement under NCLB 


4 


3 


2 


1 


Cl 95 


CI98 


c. Training or managing the 
state-level staff or school 
support teams who provide 
support to districts and 
schools identified for 
improvement under NCLB 


4 


3 


2 


1 


Cl 95 


CI98 


d. Designing or 

implementing state 
assessment and 
accountability systems 


4 


3 


2 


1 


Cl 95 


CI98 


e. Aligning state 

accountability systems 
with NCLB accountability 
systems 


4 


3 


2 


1 


Cl 95 


CI98 


f. Supporting use of 

assessment data by schools 
and districts 


4 


3 


2 


1 


C95 


C98 


g. Disseminating information 
on scientifically based 
research to districts and 
schools 


4 


3 


2 


1 


□95 


CI98 


h. Identifying and/or 

developing programs or 
models that address 
district and/or school 
needs 


4 


3 


2 


1 


Cl 95 


CI98 
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State Manager Survey 



Our state received assistance 
from the Comprehensive 
Center network with the 
following responsibilities: 


Major 

assistance 


Moderate 

assistance 


Minimal 

assistance 


No 

assistance 
at all 


NA, 

technical 
assistance 
not sought 
by the 
SEA 


NA, this 
task is not 
relevant to 
my unit’s 
work 


i. Providing training and 
other professional 
development to local 
educators in academic 
subjects (reading language 
arts, mathematics, science) 


4 


3 


2 


1 


Cl 95 


CI 98 


j. Conducting, designing, or 
overseeing parent outreach 


4 


3 


2 


1 


Cl 95 


C 98 


k. Monitoring compliance 
with NCLB requirements 
in districts and schools 


4 


3 


2 


1 


□95 


CI 98 


1. Communicating with the 
public about NCLB 
requirements or report 
cards 


4 


3 


2 


1 


Cl 95 


CI 98 


m. Other (Specify: 

) 


4 


3 


2 


1 


□95 


CI 98 
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State Manager Survey 



6. During the 2006-07 sehool year (beginning July 2006 and ending June 2007), to what 
extent did your state reeeive the following types of assistance from the Comprehensive 
Center network (your Regional Center and the Content Centers with whom your state 
has worked)? (Circle one response in each row.) 



Our state received the following 
types of assistance from the 
Comprehensive Center network: 


Major 

assistance 


Moderate 

assistance 


Minimal 

assistance 


No 

assistance 
at all 


NA, 

technical 
assistance 
not sought 
by the SEA 


a. Reviewing state plans and 
policies 


4 


3 


2 


1 


CI95 


b. Designing, delivering, or 
convening professional 
development and conferences for 
local educators 


4 


3 


2 


1 


□95 


e. Synthesizing and disseminating 
research findings 


4 


3 


2 


1 


CI95 


d. Collecting and disseminating 
information about policies and 
practices in other states 


4 


3 


2 


1 


□95 


e. Analyzing data or conducting 
needs assessments 


4 


3 


2 


1 


CI95 


f. Responding to federal planning 
and reporting requirements 


4 


3 


2 


1 


CI95 


g. Developing tools for monitoring 
programs 


4 


3 


2 


1 


CI95 


h. Facilitating work groups or 
committees 


4 


3 


2 


1 


□95 


i. Completing routine tasks more 
efficiently 


4 


3 


2 


1 


□95 


i. Other (Specify: ) 


4 


3 


2 


1 


□95 
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State Manager Survey 



Relevance and Usefulness 

7. Please eonsider all of the teehnieal assistanee that your state has reeeived from the 

Comprehensive Center network during the 2006-07 sehool year (beginning July 2006 
and ending June 2007). Considering just this set of products and services, to what degree 
was this set of activities and resources relevant to your state, in each of the following 
respects? (Circle one response in each row.) 



Comprehensive Center 
technical assistance: 


To a 
very 
high 
degree 


To a high 
degree 


To a 

moderate 

degree 


To a low 
degree 


To a very 
low degree 


Not able 
to judge 


a. Addressed a need or problem 
that the state faces 


5 


4 


3 


2 


1 


□95 


b. Addressed an important state 
priority 


5 


4 


3 


2 


1 


Cl 95 


c. Addressed a challenge that the 
state faces related to the 
implementation of NCLB 


5 


4 


3 


2 


1 


□95 


d. Responded flexibly to our 
state’s changing needs 


5 


4 


3 


2 


1 


Cl 95 


e. Provided information, advice, 
and/or resources that could be 
applied to the state’s work 


5 


4 


3 


2 


1 


Cl 95 


f Addressed the particular 
context in which our state 
operates 


5 


4 


3 


2 


1 


□95 


g. Addressed the state’s specific 
challenges (e.g., policy 
environment, leadership 
capacity, budget pressures, 
local politics) 


5 


4 


3 


2 


1 


□95 


h. Provided information, advice, 
and/or resources that could be 
used to guide decisions about 
policies, programs, and 
practices 


5 


4 


3 


2 


1 


□95 


i. Highlighted the implications of 
research findings (or 
information about best practice) 
for policies, programs, or 
practices 


5 


4 


3 


2 


1 


Cl 95 
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State Manager Survey 



8. Please eonsider all of the teehnical assistanee that your state has reeeived from the 

Comprehensive Center network during the 2006-07 sehool year (beginning July 2006 
and ending June 2007). Considering just this set of produets and serviees, to what degree 
was this set of activities and resources useful to your state, in each of the following 
respects? (Circle one response in each row.) 



Comprehensive Center 
technical assistance: 






To a 

moderate 

degree 


To a low 
degree 


To a very 
low 
degree 


Not able 
to judge 


a. Provided state staff with 
resources that were easy to 
understand and easy to use 


5 


4 


3 


2 


1 


□95 


b. Employed appropriate formats 
(e.g., work groups, conferences, 
individual consultation, written 
products) 


5 


4 


3 


2 


1 


Cl 95 


c. Provided adequate opportunity 
to learn from colleagues in 
other states 


5 


4 


3 


2 


1 


Cl 95 


d. Included adequate follow-up to 
support the use of new 
information and resources 


5 


4 


3 


2 


1 


O' 

□ 


e. Was timely 


5 


4 


3 


2 


1 


□95 


f Helped the state to solve a 
problem 


5 


4 


3 


2 


1 


O' 

□ 


g. Helped the state to maintain or 
change a policy or practice 


5 


4 


3 


2 


1 


O' 

□ 


h. Helped the state take the next 
step in a longer-term 
improvement effort 


5 


4 


3 


2 


1 


□95 


i. Provided state staff with 
information or resources that 
they will use again 


5 


4 


3 


2 


1 


□95 


j. Helped state staff to develop 
skills that they will be able to 
exercise again 


5 


4 


3 


2 


1 


Cl 95 
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State Manager Survey 



9. How fully has the teehnieal assistanee your state has reeeived from the Comprehensive 

Center network during the 2006-07 sehool year served the state’s purposes for teehnioal 

assistance? (Circle one.) 

a. It served the state’s purposes completely (Skip to question 11) 

b. It was a good start 

c. It was a start, but some important priorities were not addressed 

d. For the most part, it did not serve the state’s purposes 

10. If the Comprehensive Center network assistance during the 2006-07 school year has been 

less helpful than it might be, what were the reasons? (Circle all that apply.) 

a. The state’s most important priorities for assistance fall outside the Comprehensive 
Center’s scope of work 

b. The Comprehensive Center does not have the expertise the state needs 

c. Comprehensive Center staff are not able to spend as much time working with the state 
as we would like 

d. The process of negotiating a work scope and organizing projects takes too long 

e. The state has been unable to develop a productive working relationship with the 
Center 

f The state secures most of the technical assistance it needs from other sources 

g. The state would prefer to locate and contract directly with the experts or consultants 
from whom it needs assistance, rather than working through the Comprehensive 
Center network 

h. A policy or priority shift at the state level caused the Center’s assistance to be less 
helpful than it might 

i. Other (Specify ) 

1 1 . How could the technical assistance your state receives from the Comprehensive Center 
network be improved or made more useful to your state? 
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State Manager Survey 



Comprehensive Center Assistance Compared with Assistance from Other 
Sources 

12. Compared with the technical assistance your state has received from other sources 
(see list of sourees in Question 2), how would you rate the usefulness of the technieal 
assistanee your state has reeeived from the Comprehensive Center network for eaeh of 
the following purposes? (Circle one response in each row.) 





Much 

more 

useful 

thau 

assistauce 
from other 
sources 


Somewhat 

more 

useful 


About the 
same 


Somewhat 
less useful 


Much less 
useful 
thau 

assistauce 
from other 
sources 


Not able to 
judge 


NA, the 
state has 
not sought 
assistance 
for this 
purpose 


a. Formulating or 

refining state policies 
to respond to NCLB 
requirements 


5 


4 


3 


2 


1 


□ 95 


Cl 98 


b. Building or managing 
a statewide system of 
support for districts 
and schools identified 
for improvement 
under NCLB 


5 


4 


3 


2 


1 


□95 


C98 


e. Training or managing 
the state-level staff or 
school support teams 
who provide support 
to districts and 
schools identified for 
improvement under 
NCLB 


5 


4 


3 


2 


1 


1^95 


C98 


d. Designing or 

implementing state 
assessment and 
accountability 
systems 


5 


4 


3 


2 


1 


□95 


Cl 98 


e. Aligning state 
accountability 
systems with NCLB 
accountability 
systems 


5 


4 


3 


2 


1 


1^95 


C98 


f. Supporting use of 
assessment data by 
schools and districts 


5 


4 


3 


2 


1 


Cl 95 


Cl 98 
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State Manager Survey 





Much 

more 

useful 

thau 

assistauce 
from other 
sources 


Somewhat 

more 

useful 


About the 
same 


Somewhat 
less useful 


Much less 
useful 
thau 

assistauce 
from other 
sources 


g. Disseminating 
information on 
scientifically based 
research to districts 
and schools 


5 


4 


3 


2 


1 


h. Identifying and/or 
developing programs 
or models that address 
district and/or school 
needs 


5 


4 


3 


2 


1 


i. Providing training and 
other professional 
development to local 
educators in academic 
subjects (reading 
language arts, 
mathematics, science) 


5 


4 


3 


2 


1 


j . Conducting, 
designing, or 
overseeing parent 
outreach 


5 


4 


3 


2 


1 


k. Monitoring 

compliance with 
NCLB requirements 
in districts and 
schools 


5 


4 


3 


2 


1 


1. Communicating with 
the public about 
NCLB requirements 
or report cards 


5 


4 


3 


2 


1 















Not able to 
judge 


NA, the 
state has 
not sought 
assistance 
for this 
purpose 


1^95 


C98 


□95 


Cl 98 


□95 


Cl 98 


Cl 95 


Cl 98 


□95 


Cl 98 


Cl 95 


Cl 98 


Cl 95 


Cl 98 



m. Other (Specify: 
) 



5 



3 



2 



1 





























































State Manager Survey 



State Capacity to Carry Out Responsibilities Related to NCLB 

13. To what extent has assistanee from the Comprehensive Center network (your Regional 
Center and the Content Centers with whom your state has worked) expanded your state’s 
eapacity to earry out state responsibilities related to NCLB? (Circle one response in 
each row.) 



State Responsibilities 


To a very 
great 
extent 


To a 
great 
extent 


To a 

moderate 

extent 


To a 
small 
extent 


To a 
very 
small 
extent 


Too soon to 
tell 


NA, state 
has not 
sought 
assistance 
for this 
purpose 


a. Formulating or refining state 
policies to respond to NCLB 
requirements 


5 


4 


3 


2 


1 


□95 


Cl 98 


b. Building or managing a 
statewide system of support 
for districts and schools 
identified for improvement 
under NCLB 


5 


4 


3 


2 


1 


□95 


Cl 98 


c. Training or managing the 
state-level staff or school 
support teams who provide 
support to districts and 
schools identified for 
improvement under NCLB 


5 


4 


3 


2 


1 


□95 


Cl 98 


d. Designing or implementing 
state assessment and 
accountability systems 


5 


4 


3 


2 


1 


□95 


Cl 98 


e. Aligning state accountability 
systems with NCLB 
accountability systems 


5 


4 


3 


2 


1 


□95 


Cl 98 


f. Supporting use of 

assessment data by schools 
and districts 


5 


4 


3 


2 


1 


□95 


Cl 98 


g. Disseminating information 
on scientifically based 
research to districts and 
schools 


5 


4 


3 


2 


1 


□95 


Cl 98 


h. Identifying and/or 

developing programs or 
models that address district 
and/or school needs 


5 


4 


3 


2 


1 


□95 


C98 


i. Providing training and other 
professional development to 
local educators in academic 
subjects (reading language 
arts, mathematics, science) 


5 


4 


3 


2 


1 


□95 


C98 



D-17 










































































State Manager Survey 



State Responsibilities 


To a very 
great 
extent 


To a 
great 
extent 


To a 

moderate 

extent 


To a 
small 
extent 


To a 
very 
small 
extent 


Too soon to 
tell 


NA, state 
has not 
sought 
assistance 
for this 
purpose 


j . Conducting, designing, or 
overseeing parent outreach 


5 


4 


3 


2 


1 


1^95 


Cl 98 


k. Monitoring compliance with 
NCLB requirements in 
districts and schools 


5 


4 


3 


2 


1 


□95 


Cl 98 


1. Communicating with the 
public about NCLB 
requirements or report cards 


5 


4 


3 


2 


1 


□95 


Cl 98 


m. Other (Specify: 
) 


5 


4 


3 


2 


1 


O' 

□ 


Cl 98 



Your Job Responsibilities 

14. Please deseribe the funetions of the offiee/division/department that you direet. (Select all 
that apply.) 



a. 


School Improvement 




b. 


Curriculum and Instruction 




c. 


Assessment and Accountability 




d. 


Special education 




e. 


Federal programs (Specify: 


) 


f 


Other (Specify: 


) 



15. Which of the following statements describe your job responsibilities related to managing 
technical assistance resources from outside your organization? (Circle all that apply.) 

a. Seeking out technical assistance providers for my organization 

b. Identifying needs and priorities that will be addressed by technical assistance 

c. Negotiating a scope of work with technical assistance providers 

d. Managing others within my office/division/unit in their use of technical assistance 
services 

e. Serving as point of contact or manager for specific technical assistance projects 
f Participating in specific technical assistance projects 

g. Other (Specify: ) 

h. None, I am not responsible for managing technical assistance resources 
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State Manager Survey 



16. Did you eollaborate with any of your colleagues on your response to this survey? (Circle 
one.) 

(We will refrain from following up with your colleagues if you collaborated on this 
response.) 

a. Yes, I collaborated on this response with colleagues 

b. No, I completed this survey without consulting colleagues 



Thank you! 



Please return this survey using the self-addressed, stamped envelope enclosed 
to [name] of Policy Studies Associates at: 



[address] 




Appendix E 

Ratings of Quaiity, Reievance, and Usefuiness 




Ratings of Quality, Relevance, and Usefulness 



This appendix section presents the expert panel review rating materials and the 
participant surveys that were collected to determine ratings of technical quality, relevance, and 
usefulness. 



Expert Panel Reviewer Scoring Booklet Used for Quality Ratings 

This appendix section presents the Scoring Booklet that was used by the expert panel 
reviewers. The goal in developing the scoring rubric was to provide uniform, objective criteria 
for rating technical quality. Because the evaluation of each project was based on the professional 
judgment of three panelists, differences among raters were inevitable. Providing a well- 
developed scoring rubric and training the panelists on the use of the scoring criteria was intended 
to maximize interrater agreement and reduce bias. 

For the purposes of this evaluation, technical quality of Comprehensive Center projects 
was demonstrated through the comprehensive and balanced use of scientifically based research 
and the applicable nonresearch knowledge base. The best available research or knowledge base 
was expected to inform the delivery of technical assistance. Where rigorous and consistent 
evidence was lacking, there had to be an acknowledgment in the technical assistance provided of 
the lack of conclusive evidence and advice, and recommendations had to be appropriately 
tempered. Professional wisdom was expected to be integrated with the best available empirical 
evidence in planning for and delivering products and services. To be rated high quality, the 
materials were expected to be accurate, complete, and clear and should support use and 
implementation of the content. 



Relevance and Usefulness Ratings from RCC and State-Level Participant Surveys 

This appendix section presents the two forms of the Participant Surveys (starting on page 
E-1 1 of this appendix) that were administered to RCC and state-level staff to obtain client views 
of technical assistance from the Comprehensive Centers, particularly in the areas of the relevance 
and usefulness of Center products. 

The evaluation team developed the two parallel survey forms included in this appendix. 
One version of the survey was written using text appropriate for state-level staff who participated 
in any Center’s project. The second, similar version of the survey was written with wording 
appropriate for RCC staff who participated in a CC project. Both surveys asked questions in each 
of the following areas: 

• Project Participation; 

• Relevance and Usefulness; 

• Priorities for Technical Assistance; 

• Capacity to Carry out Responsibilities Related to NCLB, and 

• Job Responsibilities. 
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National Evaluation of the Comprehensive Centers 
Scoring Booklet for Expert Reviewers 



Fall 2007 




research 
evotuation 
lechnicol ossisTonce 

Branch Associalcs, liic. 



Policy Studies Associates 






Dimension 1: Demonstrated Use of Appropriate Knowledge Base 



Indicators 


Definitions for Indicators 


Evidence and Applicability to Project 


1A. The knowledge base 
used is appropriately 
comprehensive, given the 
project’s purpose. 


• The most important research and 
knowledge are used when 
applicable and appropriate for 
the project. 

• Appropriate legal and regulatory 
guidelines are used when 
applicable for the project. 




1B. The knowledge base 
used is accurately 
described. 


• The applicable knowledge base 
is accurately described in the 
project’s materials. 

• Research findings, regulatory 
guidelines, or professional 
wisdom used in the project 
materials are accurately 
described. 




1C. The knowledge base 
used for the project is 
relevant. 


• The research/knowledge base 
used is relevant to the topic. 

• Legal and regulatory guidelines 
that are presented are relevant to 
the topic. 
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Scoring Booklet for Quality Ratings 



Dimension 1: Demonstrated Use of Appropriate Knowledge Base (Continued) 



Indicators 


Definitions for Indicators 


Evidence and Applicability to Project 


1D. The knowledge base 
represented in the project 
is balanced, when 
appropriate. 


• More than one point of view is 
presented in the knowledge base 
used, if it is applicable and 
important. 

• Where the level of evidence 
warrants, the knowledge base 
used reflects the variety of valid 
perspectives appropriate for 
consideration in the field. 




1E. Prominence in 
findings and 

recommendations is given 
to current and best 
available scientifically 
based research, 
knowledge base, and 
professional wisdom 


• Where available, prominence is 
given to research that is 
scientifically based. 

• Research and knowledge base 
used and reflected in the project 
represent the latest generation of 
findings. 

• Research studies that used 
appropriate designs, 
methodologies, and measures 
are given prominence over those 
with weaker designs, when 
available. 




1F. Limitations in the 
knowledge base are 
acknowledged in the 
project. 


• Materials clearly note the 
strength (or limitations) of the 
knowledge or research base. 

• Findings or recommendations 
based on best practice only 
(rather than more empirical 
evidence) are accurately 
identified as such in the 
materials. 







Criteria for Scoring — Demonstrated Use of Appropriate Knowledge Base 

Directions: Circle the number that best describes the demonstrated use of the appropriate knowledge base in the project, considering the project’s performance on the various indicators identified. 




Very High Quality 


High Quality 


Moderate Quality 


Low Quality 


Very Low Quality 


5 


4 


3 


2 


1 


1A 


The research/knowledge base 
used or referenced in project 
materials includes an appropriate 
set of important sources in this 
field, pertinent to the project’s 
purpose. 




There is a research/knowledge base used or 
referenced in project materials, pertinent to the 
project’s purpose, but some important sources are not 
included. 




There is no research/knowledge base 
used or referenced in project 
materials, or the research/knowledge 
base used or referenced in project 
materials includes none of the 
important source materials in the field 
pertinent to the project’s purpose. 


1B 


All of the research findings, legal 
and regulatory guidelines, or 
professional wisdom used in the 
project’s materials is accurately 
described. 




Some inaccuracies exist in the description of the 
research/knowledge base in the project’s materials. 




Inaccuracies exist throughout the 
project in describing the 
research/knowledge base. 


1C 


All of the information presented 
about the research/ knowledge 
base in project materials is 
relevant to the topic. 




Some of the research and/or knowledge base 
reflected in the materials is relevant; however, some 
of the material presented is not relevant. 




Most of the information presented for 
the research/knowledge base is not 
relevant to the topic. 


ID 


The research/ 
knowledge base used or 
referenced contains a balanced 
variety of valid perspectives in 
this field. 




Some balance is provided in presenting the variety of 
valid perspectives in the field; however, some 
imbalance is evident. 




No balance is provided in presenting 
the various valid perspectives in the 
field. 


IE 


The most current and rigorous 
research and knowledge 
available is given the most 
prominence in project materials. 




The most current and rigorous research/knowledge 
base available is only partially reflected and given 
prominence in the project. 




The most current and rigorous 
research/knowledge base available is 
minimally reflected and given little or 
no prominence in the project. 


IF 


Limitations in the available 
research/knowledge base are 
clearly described and 
acknowledged in project 
materials. 




Limitations in the available research/knowledge base 
are only partially described and acknowledged in the 
project materials, although they exist. 




No limitations in the available 
research/knowledge base are 
acknowledged in the project 
materials, although they exist. 





Dimension 2: Fidelity of Application of the Knowledge Base to the Products and Services Provided 



Indicators 


Definitions for Indicators 


Evidence and Applicability to Project 


2A. The application of the 
research/ knowledge 
base is clear and 
accurate. 


• Accurate application of the core 
findings from the available 
knowledge base is evident in the 
project’s products, assistance, or 
advice. 

• The project’s products, 
assistance, and advice clearly 
interpret and apply the research 
and/or knowledge base. 




2B. There is consistency 
between the strength of 
the research/knowledge 
base and its proposed 
application. 


• The certainty and strength of 
recommendations in the project’s 
products, assistance, and advice 
are consistent with and 
appropriate for the level of rigor 
and certainty in the available 
research/knowledge base. 




2C. Appropriate 
emphasis is given in 
application of the most 
rigorous and consistent 
research and knowledge 
base. 


• Prominence is given to products, 
assistance, and advice derived 
from the most rigorous research 
and knowledge base. 

• Prominence is given to products, 
services, and advice derived 
from the most consistent 
research and knowledge base. 




2D. Application of 
nonempirical research 
and professional wisdom 
only are appropriately 
tempered. 


• Products, assistance, and 
advice based on a weak 
research base, limited legal or 
regulatory guidance, or primarily 
nonempirical professional 
wisdom are appropriately 
tempered. 
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Criteria for Scoring — Fidelity of Application of the Knowledge Base to the Products and Services Provided 

Directions: Circle the number that best describes the fidelity of application of the knowledge base to the products and services provided, considering the project’s performance on the various 
indicators identified. 




Very High Quality 


High Quality 


Moderate Quality 


Low Quality 


Very Low Quality 


5 


4 


3 


2 


1 


2A 


The products, assistance and 
advice provided through this 
project are consistently accurate 
and clear in interpreting and 
applying the available research 
and knowledge base used in the 
project. 




The products, assistance, and advice provided 
through this project are accurate and clear in 
interpreting and applying the available research 
and knowledge base in the project used in 
some respects but not others. 




The products, assistance and 
advice provided through this 
project are consistently 
inaccurate and unclear in 
interpreting and applying the 
available research and/or 
knowledge base used in the 
project. 


2B 


The products, advice, and 
assistance provided through this 
project are fully consistent with 
the available research and 
knowledge base. 




The products, advice, and assistance provided 
through this project are partially consistent with 
the available research and knowledge. 




The products, advice, and 
assistance provided through this 
project are not consistent and for 
the most part conflicts with the 
available research and 
knowledge base. 


2C 


The products, assistance, and 
advice provided through this 
project completely and 
consistently emphasize the 
application of findings derived 
from rigorous and consistent 
research and knowledge over 
the application of findings from 
less rigorous or consistent 
research or knowledge. 




Some of the products, assistance, and advice 
provided through this project emphasize the 
application of findings derived from rigorous and 
consistent research and knowledge over the 
application of findings from less rigorous or less 
consistent research or knowledge in some 
instances, but some products, assistance, and 
advice provided emphasize less rigorous or 
consistent research and knowledge. 




The products, assistance and 
advice provided through this 
project consistently emphasize 
the application of findings 
derived from less rigorous and 
less consistent research and 
knowledge over the application 
of findings from more rigorous or 
consistent research or 
knowledge. 


2D 


Products, assistance, and 
advice provided through this 
project that are based on weak 
research or knowledge base are 
consistently acknowledged and 
appropriately tempered. 




Products, assistance, and advice provided 
through this project that are based on weak 
research or knowledge base are partially 
acknowledged and are tempered only in part. 




Products, assistance, and advice 
provided through this project that 
are based on weak research or 
knowledge base are never 
acknowledged as such and are 
rarely if ever tempered. 





Dimension 3: Clear and Effective Delivery 



Indicators 


Definitions for Indicators 


Evidence and Applicability to Project 


3A. Information 
communicated in 
the project 
products and 
services meets the 
project’s purpose. 


• Information contained in the project 
materials is clear in meeting the 
purpose of the project. 

• Information contained in the project 
materials is complete and applicable in 
meeting the purpose of the project. 




3B. The project 

contains 

meaningful 

learning 

experiences, 

appropriate for the 

intended audience. 


• Where appropriate, products and 
services are designed to engage 
participants in effective learning 
experiences. 

• In the opinion of the reviewer, the 
audience should be interested in the 
information and recommendations 
because of the way in which they are 
delivered. 




3C. The products 
and services in the 
project are 
appropriate for the 
intended audience. 


• The products and services as presented 
are relevant and well-suited for the 
intended audience. 

• The products and services as presented 
appear to be useful for the intended 
audience. 




3D. Ideas are 

effectively 

communicated. 


• Clear and accessible language is used 
in project materials. 

• Project materials are well-written. 

• Information conveyed will be 
understood by intended audience. 
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Criteria for Scoring — Clear and Effective Delivery 



Directions: Circle the number that best describes clear and effective delivery considering the project’s performance on the various indicators identified. 





Very High Quality 


High Quality 


Moderate Quality 


Low Quality 


Very Low Quality 


5 


4 


3 


2 


1 


3A 


Project materials and the 
information presented are 
complete and applicable for 
meeting the project’s 
purpose. 




Project materials and 
information are partially 
incomplete or not applicable for 
meeting the project’s purpose. 




Project materials and 
information is incomplete or 
not at all applicable to the 
purpose of the project. 


3B 


The project materials will 
engage participants or users 
in meaningful learning 
experiences or will be of 
interest to them because of 
the way in which materials 
are delivered or packaged. 




Some of the project materials 
will engage participants in 
learning experiences or will be 
of interest to them, but some of 
the materials will not because of 
the way they are delivered or 
packaged. 




The project materials lack 
meaningful learning 
experiences and are very 
unlikely to engage or interest 
participants or users because 
of the way in which they are 
delivered or packaged. 


3C 


The products and services 
produced for this project are 
very appropriate for the 
intended audience. 




A portion of the products and 
services produced for this 
project are not appropriate or 
relevant for the intended 
audience. 




The products and services 
produced for this project are 
not well-suited (neither 
appropriate nor relevant) for 
the intended audience. 


3D 


Ideas and information are 
effectively communicated 
throughout all of the 
materials in the project. 




The ideas and information in the 
project materials are not 
effectively communicated and 
are confusing in some 
instances. 




The ideas and information 
are not effectively 
communicated throughout 
the project materials. 





Scoring Booklet for Quality Ratings 



National Evaluation of Comprehensive Centers 
2007-08 



Score Reporting Form for Expert Reviews 



Project Title: 



Comprehensive Center 
Name: 

Please list your scores helow: 

Dimension 1 Score: 

Dimension 2 Score: 

Dimension 3 Score: 



I attest that the scores listed on this scoring sheet are based on my thorough review and 
ohjective assessment of the project cover sheet and project artifacts based on the scoring 
guidance and rubric provided by the evaluation contractor. I attest that I have not 
discussed these materials or this review with anyone other than the designated evaluation 
team contact prior to submission. 

I have also provided a bulleted summary of this project’s strengths and weaknesses. 

I further attest that I do not have any undisclosed conflict of interest for this particular 
project. 

Your Name 

(Print): 

Your Signature: 

Date: 
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RCC-Level Participant Survey 



Comprehensive Center Evaluation Client Survey - RCC 

Participants 

This survey is designed to gather your feedbaek on a set of technieal assistanee aetivities 
and resourees. Your name was ineluded in a list of partieipants in one or more of the activities 
assoeiated with the projeet in the box below. The speeifie aetivities are deseribed on the yellow 
sheet in the front of this booklet. 



Per the Edueation Scienees Reform Act of 2002, Title I, Part E, Section 183, responses to 
this data eolleetion will be used only for statistieal purposes. The reports prepared for this study 
will summarize findings aeross the sample and will not assoeiate responses with a speeifie 
organization or individual. We will not provide information that identifies you or your 
organization to anyone outside the study team, except as required by law. 



Please return this survey using the self-addressed, stamped envelope enclosed 
to [name] of Policy Studies Associates (PSA) at: 

[address] 

@ 



Thank you for your feedbaek! 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid OMB control number. The valid OMB control number for this survey is 1850- 
0823. The time required to complete this survey is estimated to average 20 minutes per response, including the time 
to review instructions, search existing data sources, gather the data needed, and respond to the survey questions. 
Send comments regarding this burden estimate or any other aspect of this collection of information, including 
suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 
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Participation 

1 . Did you participate in any of the activities described on the previous page? (Circle one.) 

a. Yes 

b. No (STOP HERE - RETURN SURVEY TO [name] AT PSA) 

c. Unsure/Don’t remember (STOP HERE - RETURN SURVEY TO [name] AT PSA) 



2. Did you work for a Comprehensive Center during the period from July 2006 through June 2007? 

(Circle one.) 

a. Yes 

b. No (STOP HERE. Call [name] at -xxx-xxx-xxxx for alternate survey form.) 

3. Of this set of activities and resources (described at the beginning of the survey), how 
much time did you spend participating in each of the following types of activities or 
making use of each of the following types of resources? (Circle one response in each 
row.) 





More than 
5 days 


3-5 days 


1-2 days 


Less than 1 
day 


Not 

applicable; 
not part of 
this set of 
activities 
and 

resources 


a. Conferences 


4 


3 


2 


1 


CI 98 


b. Training 


4 


3 


2 


1 


CI 98 


c. Task force meetings 


4 


3 


2 


1 


CI 98 


d. Reviewing general or 
background information 
provided by the Content 
Center 


4 


3 


2 


1 


CI 98 


e. Using tools and other 
resources provided by 
the Content Center 


4 


3 


2 


1 


CI 98 


f Advance planning 


4 


3 


2 


1 


CI 98 


g. Ongoing consultation 
on this topic 


4 


3 


2 


1 


CI 98 


h. Follow-up and action 
plans 


4 


3 


2 


1 


Cl98 
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RCC-Level Participant Survey 



4. Were you personally involved in determining the goals or designing the eontent or 

format of the activities or resourees deseribed at the beginning of this survey? In what 
ways? (Circle all that apply,) 

a. Identifying the problem or need to be addressed 

b. Seleeting or framing the eontent 

e. Providing data or other baekground information during the planning phase 

d. Identifying or recruiting project participants 

e. Identifying or reeruiting presenters or resourees 
f Designing aetivities 

g. Planning for or leading dissemination of new ideas and information 

h. Coordinating this project with other work that my organization does 

i. Planning logisties 

j. Other (Speeify: ) 

k. / did not contribute at all to the design of this set of activities and resources 
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Relevance and Usefulness 

5. Based on your experience, to what degree was this set of aetivities and resourees 

(deseribed at the beginning of this survey) relevant to your work, in each of the following 
respeets? (Circle one response in each row.) 



This set of activities and 
resources: 


To a 
very 
high 
degree 


To a high 
degree 


To a 

moderate 

degree 


To a low 
degree 


To a very 
low degree 


Not able 
to judge 


a. Addressed a need or problem 
that my Center’s state-level 
clients face 


5 


4 


3 


2 


1 


□95 


b. Addressed an important priority 
of my Center’s state-level 
clients 


5 


4 


3 


2 


1 


□95 


c. Addressed a challenge that my 
Center’s state-level clients face 
related to the implementation of 
NCLB 


5 


4 


3 


2 


1 


□95 


d. Provided information, advice, 
and/or resources that could be 
applied to my Center’s work or 
our state-level clients’ work 


5 


4 


3 


2 


1 


□95 


e. Addressed the particular state 
context in which our state-level 
clients operate 


5 


4 


3 


2 


1 


□95 


f. Addressed our state-level 

clients’ specific challenges (e.g., 
policy environment, leadership 
capacity, budget pressures, local 
politics) 


5 


4 


3 


2 


1 


□95 


g. Provided information, advice, 
and/or resources that could be 
used by state-level clients to 
guide decisions about policies, 
programs, and practices 


5 


4 


3 


2 


1 


□95 


h. Highlighted the implications of 
research findings (or 
information about best practice) 
for state-level clients’ policies, 
programs, or practices 


5 


4 


3 


2 


1 


□95 
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RCC-Level Participant Survey 



6. Based on your experience, to what degree was this set of aetivities and resourees 
(deseribed at the beginning of this survey) useful to you, in eaeh of the following 
respeets? (Circle one response in each row.) 



This set of activities and 
resources: 


To a 
very 
high 
degree 


To a high 
degree 


To a 

moderate 

degree 


To a low 
degree 


To a very 
low degree 


Not able 
to judge 


a. Provided resources that were 
easy to understand and easy to 
use 


5 


4 


3 


2 


1 


□95 


b. Employed an appropriate format 
(e.g., a work group, a 
conference, individual 
consultation, written products) 


5 


4 


3 


2 


1 


□95 


c. Provided adequate opportunity 
to learn from other states 


5 


4 


3 


2 


1 


□95 


d. Included adequate follow-up to 
support the use of new 
information and resources 


5 


4 


3 


2 


1 


□95 


e. Were timely 


5 


4 


3 


2 


1 


□95 


f. Helped my Center to help our 
state-level clients solve a 
problem 


5 


4 


3 


2 


1 


□95 


g. Helped my Center to help our 
state-level clients maintain or 
change a policy or practice 


5 


4 


3 


2 


1 


□95 


h. Helped my Center to help our 
state-level clients take the next 
step in a longer-term 
improvement effort 


5 


4 


3 


2 


1 


□95 


i. Provided my Center with 

information or resources that we 
will use again 


5 


4 


3 


2 


1 


□95 


j. Helped my Center to develop a 
shared expertise or knowledge- 
base 


5 


4 


3 


2 


1 


□95 


k. Helped individuals in my Center 
to develop skills that they will 
use again 


5 


4 


3 


2 


1 


□95 
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RCC-Level Participant Survey 



7. How could this set of activities and resources (described on the yellow sheet inserted into this 
survey) have been made more relevant or more useful for your Center? 



Priorities for Technicai Assistance 

8. Please consider yowr Comprehensive Center’s priorities for the technical assistance that it 

receives from outside sources. With which of the following tasks does your Center have the 

greatest need for technical assistance? (Please circle your Center’s three highest priorities 

from the list below.) 

Our center has the greatest need for technical assistance with: 

a. Helping state leaders formulate or refine policies that respond to NCLB requirements 

b. Helping states build or manage a state-level system to support districts and schools identified 
for improvement under NCLB 

c. Training or helping to train or manage the state-level staff who provide support to districts 
and schools identified for improvement under NCLB 

d. Helping state-level staff work directly with low-performing districts and schools on school 
improvement activities 

e. Helping state-level staff design or implement statewide assessment and accountability 
systems 

f. Helping state-level staff to align state accountability systems with NCLB accountability 
systems 

g. Supporting the use of assessment data by schools and districts 

h. Disseminating information on scientifically based research to states, or helping state-level 
staff disseminate information on scientifically based research to districts and schools 

i. Identifying and/or developing programs or models that address state, district, or school needs 

j. Helping state-level staff provide training and other professional development to local 
educators in academic subjects (reading language arts, mathematics, science) 

k. Helping state-level staff monitor compliance with NCLB requirements in districts and 
schools 

l. Helping state-level staff communicate with the public about NCLB requirements or report 
cards 

m. Other priorities (Specify: ) 
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9. Did the set of activities and resources (described at the beginning of this survey) address any of 
the following tasks related to NCLB implementation? (Circle “yes” or “no” for all rows.) 



This set of activities and resources addressed: 


Yes 


No 


a. Helping state leaders formulate or refine policies that respond 
to NCLB requirements 


1 


0 


b. Helping states build on or manage a state-level system to 

support districts and schools identified for improvement under 
NCLB 


1 


0 


c. Training or helping to train or manage the state-level staff who 
provide support to districts and schools identified for 
improvement under NCLB 


1 


0 


d. Helping state-level staff work directly with low-performing 
districts and schools on school improvement activities 


1 


0 


e. Helping state-level staff design or implement statewide 
assessment and accountability systems 


1 


0 


f. Helping state-level staff to align state accountability systems 
with NCLB accountability systems 


1 


0 


g. Supporting the use of assessment data by schools and districts 


1 


0 


h. Disseminating information on scientifically based research to 
states, or helping state-level staff disseminate information on 
scientifically based research to districts and schools 


1 


0 


i. Identifying and/or developing programs or models that address 
state, district, or school needs 


1 


0 


j . Helping state-level staff provide training and other professional 
development to local educators in academic subjects (reading, 
language arts, mathematics, science) 


1 


0 


k. Helping state-level staff monitor compliance with NCLB 
requirements in districts and schools 


1 


0 


1. Helping state-level staff communicate with the public about 
NCLB requirements or report cards 


1 


0 


m. Other priorities (Specify: ) 


1 


0 
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RCC-Level Participant Survey 



Capacity to Carry Out Responsibilities Related to NCLB 

10. To what extent has the set of activities and resources (described at the beginning of the survey) 
expanded the capacity of your Comprehensive Center, including your own capacity, to help 
states carry out their responsibilities related to NCLB? (Circle one response in each row.) 



This set of activities and 
resources: 


To a 
great 
extent 


To a 

moderate 

extent 


To a 
small 
extent 


Not at 

all 


Too 
early to 
tell 


Does not 
apply or 
unable 
to judge 


a. Confirmed what my Center was 
already doing to help states 


4 


3 


2 


1 


□95 


O98 


b. Helped my Center to raise 

awareness of new developments 
in fields important to states 


4 


3 


2 


1 


Cl 95 


O98 


c. Helped my Center to help states 
define or understand a problem in 
new ways 


4 


3 


2 


1 


095 


O98 


d. Helped my Center to help states 
accomplish a goal or solve a 
problem 


4 


3 


2 


1 


O95 


O98 


e. Helped my Center to help states 
improve or support an ongoing 
state program, policy, or practice 


4 


3 


2 


1 


O95 


O98 


f. Helped my Center enhance states’ 
ability to address NCLB 
requirements 


4 


3 


2 


1 


O95 


O98 


g. Helped my Center improve states’ 
ability to work with districts 


4 


3 


2 


1 


O95 


O98 


h. Helped my Center improve states’ 
ability to work with schools 
identified for improvement under 
NCLB 


4 


3 


2 


1 


O95 


O98 


i. Put my Center in touch with other 
organizations engaged in similar 
tasks 


4 


3 


2 


1 


□95 


O98 


j . Enabled my Center to carry out 
its work more effectively 


4 


3 


2 


1 


O95 


O98 


k. Helped my Center to help states 
complete NCLB-related 
applications, plans, and reports 


4 


3 


2 


1 


□95 


O98 


1. Helped my Center to help states 
make a tangible change to a 
policy or practice 


4 


3 


2 


1 


O95 


O98 


m. Other (Specify: ) 


4 


3 


2 


1 


O95 


O98 


























































































RCC-Level Participant Survey 



11 . 



How could this set of activities and resources (described at the beginning of the survey) have 
better helped to expand your organization’s capacity to help states carry out responsibilities 



related to NCLB? 



Your Job Responsibilities at the Comprehensive Center 

Please consider all of your job responsibilities when responding to the questions in this section, not just 
those most closely related to the set of activities and resources described on the first page of this survey. 



12. Please describe your role at the Comprehensive Center during the period from July 2006 through 
June 2007: (Circle all that apply.) 

a. State liaison 

b. Content or topic area specialist (Specify: ) 

c. Project-specific manager or specialist 

d. Center director or assistant director 

e. Support staff (technology, research, logistics, publishing, financial) 

f. Consultant 

g. Other (Specify: ) 
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13. During the period from July 2006 through June 2007, which of the following statements best 
describes your job responsibilities related to NCLB implementation? (Circle all that apply.) 

a. Helping state leaders formulate or refine policies that respond to NCLB 
requirements 

b. Helping states build or manage a state-level system to support districts and schools 
identified for improvement under NCLB 

c. Training or helping to train or manage the state-level staff who provide support to 
districts and schools identified for improvement under NCLB 

d. Helping state-level staff work directly with low-performing districts and schools on 
school improvement activities 

e. Helping state-level staff design or implement statewide assessment and 
accountability systems 

f Helping state-level staff to align state accountability systems with NCLB 
accountability systems 

g. Supporting the use of assessment data by schools and districts 

h. Disseminating information on scientifically based research to states, or helping 
state-level staff disseminate information on scientifically based research to districts 
and schools 

i. Identifying and/or developing programs or models that address state, district, or 
school needs 

j. Helping state-level staff monitor compliance with NCLB requirements in districts 
and schools 

k. Helping state-level staff communicate with the public about NCLB requirements or 
report cards 

l. Providing technical assistance to SEA or other state-level staff 

m. Other responsibilities (Specify: 
) 
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14. During the period from July 2006 through June 2007, what percent of your time was spent on the 
tasks you selected in Question 1 3 above, combined? (Circle one.) 

a. 0-25 percent 

b. 26-50 percent 

c. 51-75 percent 

d. 76-100 percent 

15. Have you participated in any other technical assistance provided by this Content Center, other 
than the set of activities and resources described at the beginning of the survey? (Circle one.) 

a. Yes 

b. No 

c. Unsure/Don’t remember 



Thank you! 



Please return this survey using the self-addressed, stamped envelope enclosed 
to [name] of Policy Studies Associates at: 

[address] 

@ 
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Comprehensive Center Evaluation Client Survey - State- 
Level Participants 

This survey is designed to gather your feedbaek on a set of technieal assistanee activities 
and resources. Your name was included in a list of participants in one or more of the activities 
associated with the project in the box below. The specific activities are described on the yellow 
sheet in the front of this booklet. 



Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to 
this data collection will be used only for statistical purposes. The reports prepared for this study 
will summarize findings across the sample and will not associate responses with a specific 
organization or individual. We will not provide information that identifies you or your 
organization to anyone outside the study team, except as required by law. 



Please return this survey using the self-addressed, stamped envelope enclosed 
to [name] of Policy Studies Associates (PSA) at: 

[address] 

@ 



Thank you for your feedback! 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless it displays a valid OMB control number. The valid OMB control number for this survey is 1850- 
0823. The time required to complete this survey is estimated to average 20 minutes per response, including the time 
to review instructions, search existing data sources, gather the data needed, and respond to the survey questions. 
Send comments regarding this burden estimate or any other aspect of this collection of information, including 
suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 
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Participation 



1 . Did you partieipate in any of the aetivities described on the previous page? (Circle one.) 

a. Yes 

b. No (STOP HERE - RETURN SURVEY TO [name] AT PSA) 

c. Unsure/Don’t remember (STOP HERE - RETURN SURVEY TO [name] AT 
PSA) 

2. In your experience with this set of activities and resources (described at the beginning of 
this survey), how much time did you spend participating in each of the following types of 
activities or making use of each of the following types of resources? (Circle one 
response in each row.) 



Type of activity or 
resource 


More than 
5 days 


3-5 days 


1-2 days 


Less than 1 
day 


Not 

applicable; 
not part of 
this set of 
activities 
and 

resources 


a. Conferences 


4 


3 


2 


1 


CI 98 


b. Training 


4 


3 


2 


1 


CI 98 


c. Task force meetings 


4 


3 


2 


1 


CI 98 


d. Reviewing general or 
background information 
provided by the Content 
Center 


4 


3 


2 


1 


CI 98 


e. Using tools and other 
resources provided by 
the Content Center 


4 


3 


2 


1 


CI 98 


f. Advance planning 


4 


3 


2 


1 


CI 98 


g. Ongoing consultation 
on this topic 


4 


3 


2 


1 


CI 98 


h. Follow-up and action 
plans 


4 


3 


2 


1 


Cl98 
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3. Were you personally involved in determining the goals or designing the eontent or 

format of this set of aetivities and resourees (deseribed at the beginning of this survey)? 
In what ways? (Circle all that apply,) 

a. Identifying the problem or need to be addressed 

b. Seleeting or framing the eontent 

e. Providing data or other baekground information during the planning phase 

d. Identifying or reeruiting project participants 

e. Identifying or recruiting presenters or resources 

f Designing activities 

g. Planning for or leading dissemination of new ideas and information 

h. Coordinating this set of activities with other work that my organization does 

i. Planning logistics 

j. Other (Specify: ) 

L / did not contribute at all to the design of this set of activities and resources 
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Relevance and Usefulness 

4. Based on your experience, to what degree was this set of aetivities and resourees 

(deseribed at the beginning of this survey) relevant to your work, in eaeh of the following 
respeets? (Circle one response in each row.) 



This set of activities and 
resources: 


To a 
very 
high 
degree 


To a high 
degree 


To a 

moderate 

degree 


To a low 
degree 


To a very 
low degree 


Not able 
to judge 


a. Addressed a need or 
problem that my 
organization faces 


5 


4 


3 


2 


1 


Cl 95 


b. Addressed an important 
priority of my organization 


5 


4 


3 


2 


1 


Cl 95 


c. Addressed a challenge that 
my organization faces 
related to the 
implementation of NCLB 


5 


4 


3 


2 


1 


Cl 95 


d. Provided information, 
advice, and/or resources 
that could be applied to 
my organization’s work 


5 


4 


3 


2 


1 


Cl 95 


e. Addressed our particular 
state context 


5 


4 


3 


2 


1 


Cl 95 


f. Addressed my 

organization’s specific 
challenges (e.g., policy 
environment, leadership 
capacity, budget pressures, 
local politics) 


5 


4 


3 


2 


1 


Cl 95 


g. Provided information, 
advice, and/or resources 
that could be used to guide 
decisions about policies, 
programs, and practices 


5 


4 


3 


2 


1 


Cl 95 


h. Highlighted the 

implications of research 
findings (or information 
about best practice) for 
policies, programs, or 
practices 


5 


4 


3 


2 


1 


□95 
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State-Level Partieipant Survey 



5. Based on your experience, to what degree was this set of aetivities and resourees 
(deseribed at the beginning of this survey) useful to you, in eaeh of the following 
respeets? (Circle one response in each row.) 



The activities and resources: 


To a 
very 
high 
degree 


To a high 
degree 


To a 

moderate 

degree 


To a low 
degree 


To a very 
low degree 


Not able 
to judge 


a. Provided resources that were 
easy to understand and easy to 
use 


5 


4 


3 


2 


1 


□95 


b. Employed an appropriate format 
(e.g., a work group, a 
conference, individual 
consultation, written products) 


5 


4 


3 


2 


1 


O' 

□ 


c. Provided adequate opportunity 
to learn from colleagues in other 
states 


5 


4 


3 


2 


1 


Cl 95 


d. Included adequate follow-up to 
support the use of new 
information and resources 


5 


4 


3 


2 


1 


□95 


e. Were timely 


5 


4 


3 


2 


1 


□95 


f. Helped my organization to solve 
a problem 


5 


4 


3 


2 


1 


□95 


g. Helped my organization to 
maintain or change a policy or 
practice 


5 


4 


3 


2 


1 


Cl 95 


h. Helped my organization take the 
next step in a longer-term 
improvement effort 


5 


4 


3 


2 


1 


□95 


i. Provided my organization with 
information or resources that we 
will use again 


5 


4 


3 


2 


1 


Cl 95 


j. Helped my organization to 
develop a shared expertise or 
knowledge base 


5 


4 


3 


2 


1 


□95 


k. Helped individuals in my 
organization to develop skills 
that they will use again 


5 


4 


3 


2 


1 


□95 
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State-Level Partieipant Survey 



6. How could this set of activities and resources (described on the yellow sheet inserted into this 
survey) have been made more relevant or more useful for your organization? 



Priorities for Technicai Assistance 

7. Please consider yoMr organization ’s priorities for the technical assistance that it receives from 
outside sources. With which of the following tasks related to NCLB implementation does your 
organization have the greatest need for technical assistance? (Please circle your three highest 
priorities from the list below.) 

a. Formulating or refining state policies to respond to NCLB requirements 

b. Building or managing a statewide system of support for districts and schools identified 
for improvement under NCLB 

c. Training or managing the state-level staff or school support teams who provide support to 
districts and schools identified for improvement under NCLB 

d. Working directly with low-performing schools or districts on school improvement 
activities 

e. Designing or implementing state assessment and accountability systems 

f Aligning state accountability systems with NCLB accountability system 

g. Supporting use of assessment data by schools and districts 

h. Disseminating information on scientifically based research to districts and schools 

i. Identifying and/or developing programs or models that address district and/or school 
needs 

j. Providing training and other professional development to local educators in academic 
subjects (reading, language arts, mathematics, science) 

k. Monitoring compliance with NCLB requirements in districts and schools 

l. Communicating with the public about NCLB requirements or report cards 

m. Other priorities (Specify: ) 
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8. Did the set of activities and resources (described at the beginning of this survey) address any of 
the following tasks related to NCLB implementation? (Select “yes” or “no” for all rows.) 



This set of activities and resources addressed: 


Yes 


No 


a. Reformulating or refining state policies to respond to NCLB 
requirements 


1 


0 


b. Building or managing a statewide system of support for districts 
and schools identified for improvement under NCLB 


1 


0 


c. Training or managing the state-level staff or school support 
teams who provide support to districts and schools identified 
for improvement under NCLB 


1 


0 


d. Working directly with low-performing schools or districts on 
school improvement activities 


1 


0 


e. Designing or implementing state assessment and accountability 
systems 


1 


0 


f. Aligning state accountability systems with NCLB 
accountability systems 


1 


0 


g. Supporting use of assessment data by schools and districts 


1 


0 


h. Disseminating information on scientifically based research to 
districts and schools 


1 


0 


i. Identifying and/or developing programs or models that address 
district and/or school needs 


1 


0 


j . Providing training and other professional development to local 
educators in academic subjects (reading, language arts, 
mathematics, science) 


1 


0 


k. Monitoring compliance with NCLB requirements in districts 
and schools 


1 


0 


1. Communicating with the public about NCLB requirements or 
report cards 


1 


0 


m. Other priorities (Specify: ) 


1 


0 
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Capacity to Carry out Responsibilities Related to NCLB 

9. To what extent has the set of activities and resources described in the box on the first page 
expanded the capacity of your team, office, division, or unit to carry out its responsibilities 
related to NCLB? (Circle one response in each row.) 



This set of activities and 
resources: 


To a 
great 
extent 


To a 

moderate 

extent 


To a 
small 
extent 


Not at 

all 


Too 

early to 
tell 


Does not 
apply or 
unable 
to judge 


a. Confirmed what we were already 
doing 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


b. Raised awareness of new 

developments in fields important to 
my organization 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


c. Helped my organization define or 
understand a problem in new ways 


4 


3 


2 


1 


□95 


00 

O' 

□ 


d. Helped my organization take steps 
toward accomplishing a goal or 
solving a problem 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


e. Helped my organization maintain or 
improve an ongoing program, 
policy, or practice 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


f. Helped my organization plan or 
initiate a new program, policy, or 
practice 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


g. Enhanced my organization’s ability 
to address NCLB requirements 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


h. Improved my organization’s ability 
to work with districts 


4 


3 


2 


1 


□95 


00 

O' 

□ 


i. Improved my organization’s ability 
to work with schools identified for 
improvement under NCLB 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


j . Put my organization in touch with 
other organizations engaged in 
similar tasks 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


k. Enabled my organization to carry 
out its work more effectively 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


1. Helped my organization complete 
NCEB-related applications, plans, 
and reports 


4 


3 


2 


1 


Cl 95 


IC98 


m. Helped my organization make a 
tangible change to a policy or 
practice 


4 


3 


2 


1 


Cl 95 


00 

O' 

□ 


n. Other (Specify: 


4 


3 


2 


1 


□95 


00 

O' 

□ 





























































































State-Level Partieipant Survey 



10. How could this set of activities and resources have better helped to expand the capacity of your 
team, office, division, or unit to carry out responsibilities related to NCLB? 



Your Job Responsibilities 

1 1 . Please indicate the type of agency for which you worked during the period from July 2006 
through June 2007. (Circle one.) 

a. State education agency (SEA) 

b. Governor’s office or other state agency (Specify: ) 

c. Intermediate education agency (a regional resource center or area education agency serving a 
region within the state) 

d. Institution of higher education 

e. Regional Comprehensive Center 

f. Local education agency 

g. School 

h. Other (Specify: ) 

12. Did you participate in this set of activities and resources (described at the beginning of this 
survey) because you are a member of a state-sponsored school support team or a state-level task 
force? (Circle one.) 



a. Yes 

b. No 

c. Don’t know/Not sure (Explain: ) 
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State-Level Partieipant Survey 



13. Which of the following statements best describes your job responsibilities related to NCLB 
implementation, during the period from July 2006 through June 2007? (Circle all that apply.) 

a. Formulating or refining state policies to respond to NCLB requirements 

b. Building or managing a statewide system of support for districts and schools 
identified for improvement under NCLB 

c. Training or managing the state-level staff or school support teams who provide 
support to districts and schools identified for improvement under NCLB 

d. Working directly with low-performing schools or districts on school improvement 
activities 

e. Designing or implementing state assessment and accountability systems 
f Aligning state accountability systems with NCLB accountability systems 

g. Supporting use of assessment data by schools and districts 

h. Disseminating information on scientifically based research to districts and schools 

i. Identifying and/or developing programs or models that address district and/or 
school needs 

j. Providing training and other professional development to local educators in 
academic subjects (reading, language arts, mathematics, science) 

k. Monitoring compliance with NCLB requirements in districts and schools 

l. Communicating with the public about NCLB requirements or report cards 

m. Other (Specify: ) 

14. During the period from July 2006 through June 2007, what percent of your time was spent on all 
the tasks you selected in Question 1 3 above, combined? (Circle one.) 

a. 0-25 percent 

b. 26-50 percent 

c. 51-75 percent 

d. 76-100 percent 

15. Have you participated in any other technical assistance provided by this Content Center, other 
than the set of activities and resources described at the beginning of the survey? (Circle one.) 

a. Yes 

b. No 

c. Unsure/Don’t know 



Thank you! 

Please return this survey using the self-addressed, stamped envelope enclosed 
to [name] of Policy Studies Associates at: 

[address] 

@ 
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